[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:09 dist-test-slave-qlq8 krb5kdc[23420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:09 dist-test-slave-qlq8 krb5kdc[23420](info): set up 2 sockets
Sep 26 07:33:09 dist-test-slave-qlq8 krb5kdc[23420](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 07:33:10.143839 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:10 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:10.318795 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42555
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:10.640864 23433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:10.641214 23433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:10.641369 23433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:10.648708 23433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:10.648844 23433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:10.648943 23433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:10.649044 23433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:10.649124 23433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:10.658900 23433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:42555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:10.660275 23433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:10.661602 23433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:10.673511 23438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:10.673511 23439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:10.674021 23441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:10.674248 23433 server_base.cc:1034] running on GCE node
I20260926 07:33:10.674971 23433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:10.676368 23433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:10.677577 23433 hybrid_clock.cc:648] HybridClock initialized: now 1790407990677529 us; error 42 us; skew 500 ppm
I20260926 07:33:10.677911 23433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:10 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407990, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:10.681079 23433 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:10.682668 23433 webserver.cc:460] Webserver started at http://127.22.221.126:42683/ using document root <none> and password file <none>
I20260926 07:33:10.683414 23433 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:10.683552 23433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:10.683885 23433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:10.686820 23433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "bc64f4e926924b2facaecf7afab8a407"
format_stamp: "Formatted at 2026-09-26 07:33:10 on dist-test-slave-qlq8"
server_key: "0c117a70232623e4b69be50df6990a19"
server_key_iv: "14f17eb38dd7d5c8dde378e81ad7527f"
server_key_version: "encryptionkey@0"
I20260926 07:33:10.687579 23433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "bc64f4e926924b2facaecf7afab8a407"
format_stamp: "Formatted at 2026-09-26 07:33:10 on dist-test-slave-qlq8"
server_key: "0c117a70232623e4b69be50df6990a19"
server_key_iv: "14f17eb38dd7d5c8dde378e81ad7527f"
server_key_version: "encryptionkey@0"
I20260926 07:33:10.693193 23433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:33:10.696739 23448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:10.697983 23433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:33:10.698170 23433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "bc64f4e926924b2facaecf7afab8a407"
format_stamp: "Formatted at 2026-09-26 07:33:10 on dist-test-slave-qlq8"
server_key: "0c117a70232623e4b69be50df6990a19"
server_key_iv: "14f17eb38dd7d5c8dde378e81ad7527f"
server_key_version: "encryptionkey@0"
I20260926 07:33:10.698338 23433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:10.732023 23433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:10.736124 23433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:10.736421 23433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:10.756748 23433 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:42555
I20260926 07:33:10.756778 23499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:42555 every 8 connection(s)
I20260926 07:33:10.758236 23433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:10.762154 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:10.765789 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 23433
I20260926 07:33:10.766113 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:10.768011 23413 external_mini_cluster.cc:1428] Setting key 263b505a090c09ce9cb1cf27dcb32033
I20260926 07:33:10.772573 23500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407: Bootstrap starting.
I20260926 07:33:10.775732 23500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:10.776722 23500 log.cc:824] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:10.779460 23500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407: No bootstrap required, opened a new log
I20260926 07:33:10.789088 23500 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc64f4e926924b2facaecf7afab8a407" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42555 } }
I20260926 07:33:10.789579 23500 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:10.789685 23500 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc64f4e926924b2facaecf7afab8a407, State: Initialized, Role: FOLLOWER
I20260926 07:33:10.790356 23500 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc64f4e926924b2facaecf7afab8a407" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42555 } }
I20260926 07:33:10.790660 23500 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:10.790815 23500 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:10.791003 23500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:10.793685 23500 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc64f4e926924b2facaecf7afab8a407" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42555 } }
I20260926 07:33:10.794250 23500 leader_election.cc:302] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc64f4e926924b2facaecf7afab8a407; no voters:
I20260926 07:33:10.794842 23500 leader_election.cc:288] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:10.795039 23505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:10.796530 23505 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [term 1 LEADER]: Becoming Leader. State: Replica: bc64f4e926924b2facaecf7afab8a407, State: Running, Role: LEADER
I20260926 07:33:10.797094 23505 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc64f4e926924b2facaecf7afab8a407" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42555 } }
I20260926 07:33:10.797292 23500 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 07:33:10 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:10.799782 23507 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc64f4e926924b2facaecf7afab8a407. Latest consensus state: current_term: 1 leader_uuid: "bc64f4e926924b2facaecf7afab8a407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc64f4e926924b2facaecf7afab8a407" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42555 } } }
I20260926 07:33:10.800159 23507 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:10.800631 23506 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc64f4e926924b2facaecf7afab8a407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc64f4e926924b2facaecf7afab8a407" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42555 } } }
I20260926 07:33:10.800904 23506 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:10.802608 23510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:10.811586 23510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:10.814045 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:10.771827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.22.221.126:42555)
0926 07:33:10.772762 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:10.772777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:10.774122 (+ 1345us) server_negotiation.cc:407] Connection header received
0926 07:33:10.784408 (+ 10286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:10.784439 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:10.785001 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:10.785525 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:10.788425 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:10.790462 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:10.792406 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:10.793414 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:10.793616 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:10.802171 (+ 8555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:10.802187 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:10.802213 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:10.802303 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:10.807523 (+ 5220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:10.808648 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:10.808659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:10.808668 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:10.808890 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:10.809496 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:10.809501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:10.809504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:10.810356 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:10.810757 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:10.812145 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0926 07:33:10.812620 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":404,"threads_started":1}
I20260926 07:33:10.822258 23510 catalog_manager.cc:1347] Generated new cluster ID: 45bd498070434a3388974a6eee940f03
I20260926 07:33:10.822386 23510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:10.839570 23510 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:10.841215 23510 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:10.850041 23510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc64f4e926924b2facaecf7afab8a407: Generated new TSK 0
I20260926 07:33:10.850876 23510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:10.902276 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42555
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:11.248009 23526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:11.248278 23526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:11.248457 23526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:11.255693 23526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:11.255825 23526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:11.256024 23526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:11.265506 23526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42555
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:11.266752 23526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:11.268070 23526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:11.281041 23531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:11.281095 23534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:11.281065 23532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:11.281879 23526 server_base.cc:1034] running on GCE node
I20260926 07:33:11.282503 23526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:11.283471 23526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:11.284685 23526 hybrid_clock.cc:648] HybridClock initialized: now 1790407991284630 us; error 60 us; skew 500 ppm
I20260926 07:33:11.284984 23526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:11 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:11.288061 23526 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:11.289444 23526 webserver.cc:460] Webserver started at http://127.22.221.65:36197/ using document root <none> and password file <none>
I20260926 07:33:11.290136 23526 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:11.290274 23526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:11.290591 23526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:11.293478 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "39f586444e634e56981ee10ad1068e46"
format_stamp: "Formatted at 2026-09-26 07:33:11 on dist-test-slave-qlq8"
server_key: "f7bd063a4a8daee635053076e4c2eda2"
server_key_iv: "432788817787adf60c89c87d6ce3be84"
server_key_version: "encryptionkey@0"
I20260926 07:33:11.294252 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "39f586444e634e56981ee10ad1068e46"
format_stamp: "Formatted at 2026-09-26 07:33:11 on dist-test-slave-qlq8"
server_key: "f7bd063a4a8daee635053076e4c2eda2"
server_key_iv: "432788817787adf60c89c87d6ce3be84"
server_key_version: "encryptionkey@0"
I20260926 07:33:11.299568 23526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:33:11.302903 23541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:11.304272 23526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:11.304440 23526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "39f586444e634e56981ee10ad1068e46"
format_stamp: "Formatted at 2026-09-26 07:33:11 on dist-test-slave-qlq8"
server_key: "f7bd063a4a8daee635053076e4c2eda2"
server_key_iv: "432788817787adf60c89c87d6ce3be84"
server_key_version: "encryptionkey@0"
I20260926 07:33:11.304644 23526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:11.318853 23526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:11.322695 23526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:11.322979 23526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:11.323936 23526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:11.325174 23526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:11.325276 23526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:11.325418 23526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:11.325505 23526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:11.358913 23526 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:41215
I20260926 07:33:11.358951 23653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:41215 every 8 connection(s)
I20260926 07:33:11.360250 23526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:11.361761 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 23526
I20260926 07:33:11.362012 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:11.362706 23413 external_mini_cluster.cc:1428] Setting key dd972c1060a784cc1f2f1a5ccee8c788
Sep 26 07:33:11 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:11.383279 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:11.363108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:32813 (local address 127.22.221.126:42555)
0926 07:33:11.363608 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:11.363613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:11.364728 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 07:33:11.367562 (+ 2834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:11.367566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:11.367627 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:11.367716 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:11.369752 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.370453 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.372253 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.373034 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.373178 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:11.377315 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:11.377321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:11.377324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:11.377355 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:11.380023 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:11.380959 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:11.380972 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:11.380975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:11.381115 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:11.381575 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:11.381580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:11.381583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:11.381844 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:11.382002 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:11.382670 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 07:33:11.382887 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:33:11.384352 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:11.363724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42555 (local address 127.22.221.65:32813)
0926 07:33:11.364561 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:11.364612 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:11.367276 (+ 2664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:11.367992 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:11.368008 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:11.368786 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:11.369581 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:11.369607 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.370617 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.370624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:11.372111 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:11.372122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.373228 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.373232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:11.373431 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:11.374227 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:11.374250 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:11.377074 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:11.380168 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:11.380178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:11.380196 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:11.380844 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:11.381227 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:11.381232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:11.381236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:11.381433 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:11.381968 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:11.381976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:11.382313 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 07:33:11.382576 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 07:33:11.383099 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":294,"threads_started":1}
I20260926 07:33:11.386000 23654 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42555
I20260926 07:33:11.386294 23654 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:11.387017 23654 heartbeater.cc:511] Master 127.22.221.126:42555 requested a full tablet report, sending...
I20260926 07:33:11.389212 23465 ts_manager.cc:194] Registered new tserver with Master: 39f586444e634e56981ee10ad1068e46 (127.22.221.65:41215)
I20260926 07:33:11.390623 23465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:32813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:11.406539 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42555
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:11.724069 23660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:11.724388 23660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:11.724567 23660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:11.731730 23660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:11.731884 23660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:11.732072 23660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:11.741585 23660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42555
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:11.742931 23660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:11.744272 23660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:11.757612 23665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:11.757611 23666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:11.757797 23668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:11.758939 23660 server_base.cc:1034] running on GCE node
I20260926 07:33:11.759630 23660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:11.760529 23660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:11.761791 23660 hybrid_clock.cc:648] HybridClock initialized: now 1790407991761723 us; error 52 us; skew 500 ppm
I20260926 07:33:11.762151 23660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:11 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:11.771565 23660 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:11.773149 23660 webserver.cc:460] Webserver started at http://127.22.221.66:38451/ using document root <none> and password file <none>
I20260926 07:33:11.773847 23660 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:11.773952 23660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:11.774222 23660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:11.777146 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "ad42798c3de742c8805dd307c695b62b"
format_stamp: "Formatted at 2026-09-26 07:33:11 on dist-test-slave-qlq8"
server_key: "63e5a2613ca84b69d9171c8c7271d2d3"
server_key_iv: "6a2636742111157d509248f168c269a0"
server_key_version: "encryptionkey@0"
I20260926 07:33:11.777856 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "ad42798c3de742c8805dd307c695b62b"
format_stamp: "Formatted at 2026-09-26 07:33:11 on dist-test-slave-qlq8"
server_key: "63e5a2613ca84b69d9171c8c7271d2d3"
server_key_iv: "6a2636742111157d509248f168c269a0"
server_key_version: "encryptionkey@0"
I20260926 07:33:11.782825 23660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:11.786168 23675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:11.787490 23660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:11.787613 23660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "ad42798c3de742c8805dd307c695b62b"
format_stamp: "Formatted at 2026-09-26 07:33:11 on dist-test-slave-qlq8"
server_key: "63e5a2613ca84b69d9171c8c7271d2d3"
server_key_iv: "6a2636742111157d509248f168c269a0"
server_key_version: "encryptionkey@0"
I20260926 07:33:11.787765 23660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:11.804744 23660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:11.809384 23660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:11.809629 23660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:11.810539 23660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:11.811998 23660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:11.812072 23660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:11.812160 23660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:11.812206 23660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:11.846110 23660 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:43757
I20260926 07:33:11.846148 23787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:43757 every 8 connection(s)
I20260926 07:33:11.847433 23660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:11.852699 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 23660
I20260926 07:33:11.852895 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:11.853319 23413 external_mini_cluster.cc:1428] Setting key 49cf884b16826143f33d36a6585bf8f9
Sep 26 07:33:11 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:11.867442 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:11.850007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:49769 (local address 127.22.221.126:42555)
0926 07:33:11.850203 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:11.850220 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:11.851426 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 07:33:11.853362 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:11.853368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:11.853452 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:11.853567 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:11.855511 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.856207 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.857714 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.858120 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.858247 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:11.861822 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:11.861827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:11.861830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:11.861860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:11.864475 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:11.865196 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:11.865205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:11.865208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:11.865326 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:11.865742 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:11.865747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:11.865750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:11.866014 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:11.866164 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:11.866828 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 07:33:11.867066 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:33:11.868436 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:11.850502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42555 (local address 127.22.221.66:49769)
0926 07:33:11.851269 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:11.851317 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:11.853091 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:11.853823 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:11.853836 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:11.854549 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:11.855342 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:11.855372 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.856355 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.856360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:11.857580 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:11.857591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:11.858332 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:11.858338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:11.858573 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:11.859224 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:11.859290 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:11.861608 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:11.864622 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:11.864631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:11.864648 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:11.865074 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:11.865432 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:11.865437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:11.865441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:11.865631 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:11.866134 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:11.866141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:11.866484 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 07:33:11.866743 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 07:33:11.867290 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":275,"threads_started":1}
I20260926 07:33:11.869829 23788 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42555
I20260926 07:33:11.870146 23788 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
I20260926 07:33:11.870873 23788 heartbeater.cc:511] Master 127.22.221.126:42555 requested a full tablet report, sending...
I20260926 07:33:11.872480 23465 ts_manager.cc:194] Registered new tserver with Master: ad42798c3de742c8805dd307c695b62b (127.22.221.66:43757)
I20260926 07:33:11.873284 23465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:49769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:11.894237 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42555
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:12.225085 23793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:12.225378 23793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:12.225544 23793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:12.232946 23793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:12.233064 23793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:12.233258 23793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:12.243449 23793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42555
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:12.245255 23793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:12.246753 23793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:12.260377 23798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:12.260344 23799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:12.260511 23801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:12.261175 23793 server_base.cc:1034] running on GCE node
I20260926 07:33:12.261818 23793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:12.262809 23793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:12.264035 23793 hybrid_clock.cc:648] HybridClock initialized: now 1790407992263973 us; error 55 us; skew 500 ppm
I20260926 07:33:12.264389 23793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:12.267622 23793 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:12.269094 23793 webserver.cc:460] Webserver started at http://127.22.221.67:33497/ using document root <none> and password file <none>
I20260926 07:33:12.269783 23793 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:12.269933 23793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:12.270263 23793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:12.273324 23793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "06334ceb3b1040ccb1f0ba1941c669ee"
format_stamp: "Formatted at 2026-09-26 07:33:12 on dist-test-slave-qlq8"
server_key: "5b5171f3f4409c5edf2f52cb2d918cb0"
server_key_iv: "4a8d2bb1673f6211f00eb367b2aec501"
server_key_version: "encryptionkey@0"
I20260926 07:33:12.274024 23793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "06334ceb3b1040ccb1f0ba1941c669ee"
format_stamp: "Formatted at 2026-09-26 07:33:12 on dist-test-slave-qlq8"
server_key: "5b5171f3f4409c5edf2f52cb2d918cb0"
server_key_iv: "4a8d2bb1673f6211f00eb367b2aec501"
server_key_version: "encryptionkey@0"
I20260926 07:33:12.279214 23793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:33:12.283284 23808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:12.284713 23793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:33:12.284876 23793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "06334ceb3b1040ccb1f0ba1941c669ee"
format_stamp: "Formatted at 2026-09-26 07:33:12 on dist-test-slave-qlq8"
server_key: "5b5171f3f4409c5edf2f52cb2d918cb0"
server_key_iv: "4a8d2bb1673f6211f00eb367b2aec501"
server_key_version: "encryptionkey@0"
I20260926 07:33:12.285050 23793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:12.320468 23793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:12.325127 23793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:12.325436 23793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:12.326495 23793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:12.327953 23793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:12.328040 23793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:12.328181 23793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:12.328241 23793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:12.365664 23793 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:40667
I20260926 07:33:12.365736 23920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:40667 every 8 connection(s)
I20260926 07:33:12.366884 23793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:12.372541 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 23793
I20260926 07:33:12.372738 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:12.373312 23413 external_mini_cluster.cc:1428] Setting key 717b5bd9de6ab674f50578e107bba69a
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:12.387490 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.369592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:56285 (local address 127.22.221.126:42555)
0926 07:33:12.370054 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:12.370061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:12.370996 (+ 935us) server_negotiation.cc:407] Connection header received
0926 07:33:12.373190 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:12.373194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:12.373309 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:12.373456 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:12.375490 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.376664 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.378324 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.378715 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.378821 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.382125 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:12.382130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:12.382133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:12.382165 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:12.384751 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.385433 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.385437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.385439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.385548 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.385933 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.385936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.385938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.386193 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:12.386301 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:12.386956 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 07:33:12.387119 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260926 07:33:12.388708 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.370163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42555 (local address 127.22.221.67:56285)
0926 07:33:12.370838 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:12.370873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:12.372873 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:12.373735 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:12.373748 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:12.374450 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:12.375310 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:12.375345 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.376840 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.376844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:12.378193 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:12.378205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.378903 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.378908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:12.379093 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.379798 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:12.379820 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:12.381941 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:12.384876 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:12.384882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:12.384894 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:12.385326 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:12.385640 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:12.385643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:12.385645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:12.385836 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:12.386321 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:12.386328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:12.386635 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 07:33:12.386877 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 07:33:12.387317 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":281,"threads_started":1}
I20260926 07:33:12.390166 23921 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42555
I20260926 07:33:12.390471 23921 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:12.391171 23921 heartbeater.cc:511] Master 127.22.221.126:42555 requested a full tablet report, sending...
I20260926 07:33:12.392562 23465 ts_manager.cc:194] Registered new tserver with Master: 06334ceb3b1040ccb1f0ba1941c669ee (127.22.221.67:40667)
I20260926 07:33:12.393325 23654 heartbeater.cc:503] Master 127.22.221.126:42555 was elected leader, sending a full tablet report...
I20260926 07:33:12.393379 23465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:56285
I20260926 07:33:12.401803 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:12.446717 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.431503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.22.221.65:41215)
0926 07:33:12.432168 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:12.432178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:12.432210 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:33:12.432406 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:12.432414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:12.432694 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:12.432868 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:12.434292 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.435384 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.436617 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.437182 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.437323 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.441157 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:12.441167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:12.441181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:12.441218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:12.443980 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.444645 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.444652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.444654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.444778 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.445144 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.445149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.445152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.445508 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:12.445803 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:12.446099 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:33:12.446372 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":306,"threads_started":1}
W20260926 07:33:12.448047 23569 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54192
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:12.493839 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.480061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.22.221.65:41215)
0926 07:33:12.480287 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:12.480294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:12.480558 (+ 264us) server_negotiation.cc:407] Connection header received
0926 07:33:12.480778 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:12.480782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:12.480848 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:12.480955 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:12.482269 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.482973 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.484326 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.484746 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.484893 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.488792 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:12.488799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:12.488804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:12.488844 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:12.491572 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.492263 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.492267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.492269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.492386 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.492685 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.492687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.492689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.492916 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:12.493091 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:12.493341 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:33:12.493516 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:33:12.495266 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 23526
I20260926 07:33:12.507907 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.496264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.22.221.65:41215)
0926 07:33:12.496446 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:12.496454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:12.496720 (+ 266us) server_negotiation.cc:407] Connection header received
0926 07:33:12.496905 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:12.496909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:12.497003 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:12.497110 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:12.498367 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.499029 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.500253 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.500655 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.500746 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.502982 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:12.502992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:12.502995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:12.503024 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:12.505385 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.506035 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.506040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.506042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.506163 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.506512 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.506519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.506523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.506772 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:12.507014 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:12.507353 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 07:33:12.507561 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:33:12.508651 23649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:12.591018 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 23526
I20260926 07:33:12.607421 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 23660
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:12.623271 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.608347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.22.221.66:43757)
0926 07:33:12.608863 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:12.608870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:12.608894 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:33:12.608960 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:12.608967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:12.609185 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:12.609361 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:12.610591 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.611711 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.612946 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.613508 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.613645 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.617681 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:12.617692 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:12.617706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:12.617740 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:12.620657 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.621299 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.621306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.621309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.621432 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.621806 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.621809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.621811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.622135 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:12.622351 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:12.622639 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:33:12.622828 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260926 07:33:12.624575 23783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:12.707144 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 23660
I20260926 07:33:12.723294 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 23793
Sep 26 07:33:12 dist-test-slave-qlq8 krb5kdc[23420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:12.738793 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:12.724218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.22.221.67:40667)
0926 07:33:12.724848 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:12.724855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:12.724882 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:33:12.724945 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:12.724951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:12.725173 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:12.725303 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:12.726465 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.727547 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.728745 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:12.729298 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:12.729391 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:12.733507 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:12.733518 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:12.733533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:12.733567 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:12.736300 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.736882 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.736889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.736891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.737023 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:12.737376 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:12.737379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:12.737381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:12.737701 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:12.737922 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:12.738204 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 07:33:12.738398 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":332,"threads_started":1}
I20260926 07:33:12.740167 23916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:12.823043 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 23793
I20260926 07:33:12.838410 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 23433
I20260926 07:33:12.839638 23495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:12.942514 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 23433
2026-09-26T07:33:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:13 dist-test-slave-qlq8 krb5kdc[23943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:13 dist-test-slave-qlq8 krb5kdc[23943](info): set up 2 sockets
Sep 26 07:33:13 dist-test-slave-qlq8 krb5kdc[23943](info): commencing operation
krb5kdc: starting...
W20260926 07:33:14.015205 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:14 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:14.198704 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43967
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:14.518783 23956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:14.519061 23956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:14.519147 23956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:14.526428 23956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:14.526537 23956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:14.526582 23956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:14.526643 23956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:14.526684 23956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:14.536547 23956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:14.537775 23956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:14.538966 23956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:14.550413 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:14.550386 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:14.550515 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:14.551391 23956 server_base.cc:1034] running on GCE node
I20260926 07:33:14.552211 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:14.553594 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:14.554797 23956 hybrid_clock.cc:648] HybridClock initialized: now 1790407994554751 us; error 35 us; skew 500 ppm
I20260926 07:33:14.555140 23956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:14 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:14.558334 23956 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:14.559777 23956 webserver.cc:460] Webserver started at http://127.22.221.126:33607/ using document root <none> and password file <none>
I20260926 07:33:14.560429 23956 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:14.560567 23956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:14.560881 23956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:14.563810 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "9b7f0502ba994adf97c259579e245e05"
format_stamp: "Formatted at 2026-09-26 07:33:14 on dist-test-slave-qlq8"
server_key: "0512ced5f56e7ba5a33c83ea24995d71"
server_key_iv: "1382c6837b959e61bcac4f16630bf32e"
server_key_version: "encryptionkey@0"
I20260926 07:33:14.564548 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "9b7f0502ba994adf97c259579e245e05"
format_stamp: "Formatted at 2026-09-26 07:33:14 on dist-test-slave-qlq8"
server_key: "0512ced5f56e7ba5a33c83ea24995d71"
server_key_iv: "1382c6837b959e61bcac4f16630bf32e"
server_key_version: "encryptionkey@0"
I20260926 07:33:14.569653 23956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:33:14.572942 23971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:14.574177 23956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:33:14.574352 23956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "9b7f0502ba994adf97c259579e245e05"
format_stamp: "Formatted at 2026-09-26 07:33:14 on dist-test-slave-qlq8"
server_key: "0512ced5f56e7ba5a33c83ea24995d71"
server_key_iv: "1382c6837b959e61bcac4f16630bf32e"
server_key_version: "encryptionkey@0"
I20260926 07:33:14.574535 23956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:14.588008 23956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:14.592267 23956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:14.592561 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:14.610565 23956 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:43967
I20260926 07:33:14.610594 24022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:43967 every 8 connection(s)
I20260926 07:33:14.612016 23956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:14.615437 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 23956
I20260926 07:33:14.615669 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:14.615805 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:14.616381 23413 external_mini_cluster.cc:1428] Setting key 2f38e4ffdf44518f8916a9c00eb3775b
I20260926 07:33:14.629818 24023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05: Bootstrap starting.
Sep 26 07:33:14 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:14.633778 24023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:14.635191 24023 log.cc:824] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:14.637996 24023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05: No bootstrap required, opened a new log
I20260926 07:33:14.640106 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:14.618278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.22.221.126:43967)
0926 07:33:14.619084 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:14.619099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:14.619151 (+ 52us) server_negotiation.cc:407] Connection header received
0926 07:33:14.621029 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:14.621061 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:14.621587 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:14.622052 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:14.623376 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:14.625241 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:14.626540 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:14.627640 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:14.627821 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:14.632095 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:14.632103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:14.632117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:14.632159 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:14.636134 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:14.636804 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:14.636817 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:14.636822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:14.636987 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:14.637344 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:14.637349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:14.637352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:14.637853 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:14.638184 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:14.638672 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 07:33:14.639063 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":345,"threads_started":1}
I20260926 07:33:14.646675 24023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7f0502ba994adf97c259579e245e05" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43967 } }
I20260926 07:33:14.647058 24023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:14.647163 24023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7f0502ba994adf97c259579e245e05, State: Initialized, Role: FOLLOWER
I20260926 07:33:14.647796 24023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7f0502ba994adf97c259579e245e05" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43967 } }
I20260926 07:33:14.648041 24023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:14.648185 24023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:14.648360 24023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:14.650565 24023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7f0502ba994adf97c259579e245e05" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43967 } }
I20260926 07:33:14.651127 24023 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7f0502ba994adf97c259579e245e05; no voters:
I20260926 07:33:14.651832 24023 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:14.651918 24028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:14.652669 24028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [term 1 LEADER]: Becoming Leader. State: Replica: 9b7f0502ba994adf97c259579e245e05, State: Running, Role: LEADER
I20260926 07:33:14.653242 24028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7f0502ba994adf97c259579e245e05" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43967 } }
I20260926 07:33:14.653895 24023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:14.656132 24029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b7f0502ba994adf97c259579e245e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7f0502ba994adf97c259579e245e05" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43967 } } }
I20260926 07:33:14.656351 24030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7f0502ba994adf97c259579e245e05. Latest consensus state: current_term: 1 leader_uuid: "9b7f0502ba994adf97c259579e245e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7f0502ba994adf97c259579e245e05" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43967 } } }
I20260926 07:33:14.656570 24029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:14.656646 24030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:14.658195 24037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:14.663558 24037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:14.672369 24037 catalog_manager.cc:1347] Generated new cluster ID: 1ba3a1ac60b04c86a7149d6cff97399b
I20260926 07:33:14.672511 24037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:14.681406 24037 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:14.682643 24037 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:14.693666 24037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b7f0502ba994adf97c259579e245e05: Generated new TSK 0
I20260926 07:33:14.694494 24037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:14.751509 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43967
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:15.080083 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:15.080416 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:15.080528 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:15.088379 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:15.088511 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:15.088641 24049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:15.099725 24049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:15.100996 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:15.102296 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:15.115666 24057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:15.115753 24054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:15.115732 24055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:15.116245 24049 server_base.cc:1034] running on GCE node
I20260926 07:33:15.116909 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:15.117802 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:15.119017 24049 hybrid_clock.cc:648] HybridClock initialized: now 1790407995118950 us; error 59 us; skew 500 ppm
I20260926 07:33:15.119403 24049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:15 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407995, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:15.122614 24049 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:15.124132 24049 webserver.cc:460] Webserver started at http://127.22.221.65:45879/ using document root <none> and password file <none>
I20260926 07:33:15.124814 24049 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:15.124961 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:15.125280 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:15.128280 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "3fbf82067b104668a51f3f195295b5a3"
format_stamp: "Formatted at 2026-09-26 07:33:15 on dist-test-slave-qlq8"
server_key: "e3b8b9c669239a555eb24f89393120de"
server_key_iv: "048d8968b7ee8cd28777167cda21278d"
server_key_version: "encryptionkey@0"
I20260926 07:33:15.129086 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "3fbf82067b104668a51f3f195295b5a3"
format_stamp: "Formatted at 2026-09-26 07:33:15 on dist-test-slave-qlq8"
server_key: "e3b8b9c669239a555eb24f89393120de"
server_key_iv: "048d8968b7ee8cd28777167cda21278d"
server_key_version: "encryptionkey@0"
I20260926 07:33:15.134617 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:15.138232 24064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:15.140053 24049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:15.140210 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "3fbf82067b104668a51f3f195295b5a3"
format_stamp: "Formatted at 2026-09-26 07:33:15 on dist-test-slave-qlq8"
server_key: "e3b8b9c669239a555eb24f89393120de"
server_key_iv: "048d8968b7ee8cd28777167cda21278d"
server_key_version: "encryptionkey@0"
I20260926 07:33:15.140398 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:15.165731 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:15.169796 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:15.170050 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:15.170938 24049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:15.172461 24049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:15.172537 24049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:15.172622 24049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:15.172667 24049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:15.207486 24049 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:40119
I20260926 07:33:15.207531 24176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:40119 every 8 connection(s)
I20260926 07:33:15.208791 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:15.211421 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24049
I20260926 07:33:15.211653 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:15.212363 23413 external_mini_cluster.cc:1428] Setting key c99293ec4309b07f749865a3131b0af4
Sep 26 07:33:15 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407995, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:15.231671 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:15.211423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:60851 (local address 127.22.221.126:43967)
0926 07:33:15.211971 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:15.211978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:15.213076 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 07:33:15.215514 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:15.215518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:15.215579 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:15.215668 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:15.217816 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.218552 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.220394 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.221124 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.221275 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:15.224564 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:15.224574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:15.224582 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:15.224633 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:15.228371 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:15.229103 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:15.229111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:15.229115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:15.229260 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:15.229725 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:15.229733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:15.229739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:15.230194 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:15.230378 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:15.231104 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 07:33:15.231353 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":242,"threads_started":1}
I20260926 07:33:15.232757 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:15.212076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43967 (local address 127.22.221.65:60851)
0926 07:33:15.212892 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:15.212945 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:15.215186 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:15.215950 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:15.215963 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:15.216771 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:15.217636 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:15.217662 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.218700 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.218709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:15.220245 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:15.220258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.221327 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.221331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:15.221530 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:15.222202 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:15.222224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:15.224336 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:15.228521 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:15.228528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:15.228540 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:15.228955 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:15.229380 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:15.229384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:15.229386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:15.229583 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:15.230336 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:15.230342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:15.230725 (+ 383us) client_negotiation.cc:769] Sending connection context
0926 07:33:15.230956 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 07:33:15.231410 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":317,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:33:15.234411 24177 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43967
I20260926 07:33:15.234756 24177 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:15.235683 24177 heartbeater.cc:511] Master 127.22.221.126:43967 requested a full tablet report, sending...
I20260926 07:33:15.238027 23988 ts_manager.cc:194] Registered new tserver with Master: 3fbf82067b104668a51f3f195295b5a3 (127.22.221.65:40119)
I20260926 07:33:15.240339 23988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:60851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:15.259356 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43967
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:15.569335 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:15.569597 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:15.569706 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:15.576793 24183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:15.576892 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:15.577010 24183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:15.586524 24183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:15.587960 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:15.589267 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:15.603529 24189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:15.603694 24188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:15.603718 24191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:15.604506 24183 server_base.cc:1034] running on GCE node
I20260926 07:33:15.605130 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:15.606019 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:15.607219 24183 hybrid_clock.cc:648] HybridClock initialized: now 1790407995607177 us; error 34 us; skew 500 ppm
I20260926 07:33:15.607539 24183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:15 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407995, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:15.610682 24183 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:15.612213 24183 webserver.cc:460] Webserver started at http://127.22.221.66:40561/ using document root <none> and password file <none>
I20260926 07:33:15.612881 24183 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:15.613016 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:15.613343 24183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:15.616339 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "bd747871a9d24aeeb174958d558416d5"
format_stamp: "Formatted at 2026-09-26 07:33:15 on dist-test-slave-qlq8"
server_key: "acd6c804384dab7d92124e172cb76261"
server_key_iv: "869f1133621695ff817a6e38747ac718"
server_key_version: "encryptionkey@0"
I20260926 07:33:15.617136 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "bd747871a9d24aeeb174958d558416d5"
format_stamp: "Formatted at 2026-09-26 07:33:15 on dist-test-slave-qlq8"
server_key: "acd6c804384dab7d92124e172cb76261"
server_key_iv: "869f1133621695ff817a6e38747ac718"
server_key_version: "encryptionkey@0"
I20260926 07:33:15.622516 24183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 07:33:15.625980 24198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:15.627317 24183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:15.627478 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "bd747871a9d24aeeb174958d558416d5"
format_stamp: "Formatted at 2026-09-26 07:33:15 on dist-test-slave-qlq8"
server_key: "acd6c804384dab7d92124e172cb76261"
server_key_iv: "869f1133621695ff817a6e38747ac718"
server_key_version: "encryptionkey@0"
I20260926 07:33:15.627689 24183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:15.657447 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:15.662006 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:15.662298 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:15.663323 24183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:15.664675 24183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:15.664779 24183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:15.664906 24183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:15.664986 24183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:15.698341 24183 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:45803
I20260926 07:33:15.698391 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:45803 every 8 connection(s)
I20260926 07:33:15.699724 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:15.706622 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24183
I20260926 07:33:15.706817 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:15.707316 23413 external_mini_cluster.cc:1428] Setting key 86fce22e12678157b838643d069d484b
Sep 26 07:33:15 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407995, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:15.720755 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:15.702240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:42457 (local address 127.22.221.126:43967)
0926 07:33:15.702420 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:15.702425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:15.703661 (+ 1236us) server_negotiation.cc:407] Connection header received
0926 07:33:15.705519 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:15.705524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:15.705618 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:15.705729 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:15.707335 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.708239 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.709925 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.710395 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.710518 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:15.714837 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:15.714842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:15.714845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:15.714877 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:15.717452 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:15.718376 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:15.718379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:15.718381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:15.718518 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:15.719015 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:15.719018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:15.719020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:15.719328 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:15.719440 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:15.720303 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 07:33:15.720491 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:33:15.722347 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:15.702785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43967 (local address 127.22.221.66:42457)
0926 07:33:15.703512 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:15.703549 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:15.705230 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:15.705962 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:15.705971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:15.706497 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:15.707153 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:15.707171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.708396 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.708400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:15.709776 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:15.709789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:15.710637 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:15.710653 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:15.711004 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:15.711753 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:15.711789 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:15.714590 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:15.717605 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:15.717616 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:15.717636 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:15.718258 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:15.718627 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:15.718635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:15.718641 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:15.718893 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:15.719449 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:15.719460 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:15.719910 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 07:33:15.720220 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 07:33:15.720804 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":276,"threads_started":1}
I20260926 07:33:15.723845 24311 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43967
I20260926 07:33:15.724149 24311 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:15.724925 24311 heartbeater.cc:511] Master 127.22.221.126:43967 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:33:15.726394 23988 ts_manager.cc:194] Registered new tserver with Master: bd747871a9d24aeeb174958d558416d5 (127.22.221.66:45803)
I20260926 07:33:15.727272 23988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:42457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:15.750875 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43967
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:16.066926 24316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:16.067278 24316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:16.067452 24316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:16.074577 24316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:16.074721 24316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:16.074915 24316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:16.084920 24316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43967
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:16.086227 24316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:16.087615 24316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:16.100432 24321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:16.100502 24324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:16.100390 24322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:16.101548 24316 server_base.cc:1034] running on GCE node
I20260926 07:33:16.102157 24316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:16.103119 24316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:16.104349 24316 hybrid_clock.cc:648] HybridClock initialized: now 1790407996104274 us; error 122 us; skew 500 ppm
I20260926 07:33:16.104679 24316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:16.107787 24316 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:16.109175 24316 webserver.cc:460] Webserver started at http://127.22.221.67:36191/ using document root <none> and password file <none>
I20260926 07:33:16.109807 24316 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:16.109901 24316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:16.110164 24316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:16.113029 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "f689034f9dbf4c38b04dbf6be044f541"
format_stamp: "Formatted at 2026-09-26 07:33:16 on dist-test-slave-qlq8"
server_key: "4bedf0fd5f33a02aec1aa105f3641af8"
server_key_iv: "52fb8c2d20a873520c412b238b777501"
server_key_version: "encryptionkey@0"
I20260926 07:33:16.113807 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "f689034f9dbf4c38b04dbf6be044f541"
format_stamp: "Formatted at 2026-09-26 07:33:16 on dist-test-slave-qlq8"
server_key: "4bedf0fd5f33a02aec1aa105f3641af8"
server_key_iv: "52fb8c2d20a873520c412b238b777501"
server_key_version: "encryptionkey@0"
I20260926 07:33:16.118774 24316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:16.122133 24331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:16.123624 24316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:16.123826 24316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "f689034f9dbf4c38b04dbf6be044f541"
format_stamp: "Formatted at 2026-09-26 07:33:16 on dist-test-slave-qlq8"
server_key: "4bedf0fd5f33a02aec1aa105f3641af8"
server_key_iv: "52fb8c2d20a873520c412b238b777501"
server_key_version: "encryptionkey@0"
I20260926 07:33:16.124044 24316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:16.140093 24316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:16.144416 24316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:16.144727 24316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:16.145705 24316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:16.147018 24316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:16.147121 24316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:16.147291 24316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:16.147368 24316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 07:33:16.180217 24316 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:45331
I20260926 07:33:16.180253 24443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:45331 every 8 connection(s)
I20260926 07:33:16.181593 24316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:16.187179 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24316
I20260926 07:33:16.187399 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:16.187930 23413 external_mini_cluster.cc:1428] Setting key 61c7dad775198a00c6308b2fd94e30d2
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:16.202453 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.184683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:45427 (local address 127.22.221.126:43967)
0926 07:33:16.184907 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.184913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.186159 (+ 1246us) server_negotiation.cc:407] Connection header received
0926 07:33:16.188595 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.188602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.188695 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.188800 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.190716 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.191632 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.193116 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.193511 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.193603 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.196871 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.196876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.196880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.196932 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.199611 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.200252 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.200256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.200258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.200377 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.200732 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.200735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.200737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.200985 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.201120 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.201879 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.202100 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:33:16.203480 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.185280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43967 (local address 127.22.221.67:45427)
0926 07:33:16.185986 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:16.186032 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:16.188270 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:16.189067 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:16.189075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:16.189865 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:16.190499 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.190517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.191781 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.191785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.192999 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.193010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.193688 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.193692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.193886 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.194496 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:16.194517 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:16.196686 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:16.199730 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:16.199736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:16.199746 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:16.200155 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:16.200468 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:16.200471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:16.200473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:16.200645 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:16.201116 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:16.201125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:16.201514 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 07:33:16.201779 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 07:33:16.202285 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":274,"threads_started":1}
I20260926 07:33:16.204716 24444 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43967
I20260926 07:33:16.205017 24444 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:16.205703 24444 heartbeater.cc:511] Master 127.22.221.126:43967 requested a full tablet report, sending...
I20260926 07:33:16.207069 23988 ts_manager.cc:194] Registered new tserver with Master: f689034f9dbf4c38b04dbf6be044f541 (127.22.221.67:45331)
I20260926 07:33:16.208158 23988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:45427
I20260926 07:33:16.215307 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:16.234092 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.221445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.22.221.126:43967)
0926 07:33:16.221614 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.221618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.222180 (+ 562us) server_negotiation.cc:407] Connection header received
0926 07:33:16.222304 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.222308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.222372 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.222459 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.224056 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.224795 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.226034 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.226469 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.226568 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.229044 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.229049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.229052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.229083 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.231801 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.232463 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.232467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.232469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.232616 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.232937 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.232940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.232942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.233251 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.233440 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.233711 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.233866 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:33:16.243409 24177 heartbeater.cc:503] Master 127.22.221.126:43967 was elected leader, sending a full tablet report...
I20260926 07:33:16.240192 23988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:16.247673 23988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:16.278910 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.259634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:45803 (local address 127.0.0.1:56538)
0926 07:33:16.262032 (+ 2398us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:16.262084 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:16.262294 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:16.270746 (+ 8452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:16.270753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:16.270793 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:16.271351 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.271362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.273483 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.273489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.276066 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.276079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.277839 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.277846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.278176 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.278202 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:33:16.278276 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:33:16.278488 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"mutex_wait_us":232,"thread_start_us":9015,"threads_started":1}
I20260926 07:33:16.278990 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.260492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.22.221.67:45331)
0926 07:33:16.269762 (+ 9270us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.269774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.269818 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:33:16.269952 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.269963 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.270259 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.271308 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:16.271647 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.273111 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.276261 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.277569 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.277715 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.277978 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.278475 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.278633 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9032,"thread_start_us":8845,"threads_started":1}
I20260926 07:33:16.278988 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.259871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.22.221.66:45803)
0926 07:33:16.270075 (+ 10204us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.270087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.270120 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:33:16.270235 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.270244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.270495 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.271457 (+ 962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:16.271690 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.273340 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.276231 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.277598 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.278180 (+ 582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.278373 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.278457 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.278638 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10043,"thread_start_us":9900,"threads_started":1}
I20260926 07:33:16.279014 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.269297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.22.221.65:40119)
0926 07:33:16.270421 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.270429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.271200 (+ 771us) server_negotiation.cc:407] Connection header received
0926 07:33:16.271480 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.271487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.271703 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.271890 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:16.272755 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.274040 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.276007 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.277588 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.277735 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.277963 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.278565 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.278702 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":812,"threads_started":1}
I20260926 07:33:16.279896 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.270443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40119 (local address 127.0.0.1:45244)
0926 07:33:16.271041 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:16.271059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:16.271255 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:16.272144 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:16.272149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:16.272172 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:16.272521 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.272532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.274185 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.274191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.275846 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.275858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.278117 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.278124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.278285 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.278316 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:33:16.279276 (+ 960us) client_negotiation.cc:240] Negotiation successful
0926 07:33:16.279416 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":55,"thread_start_us":301,"threads_started":1}
I20260926 07:33:16.279939 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.260930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:45331 (local address 127.0.0.1:39522)
0926 07:33:16.268961 (+ 8031us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:16.268982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:16.269088 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:16.270495 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:16.270506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:16.270542 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:16.271050 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.271060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.273268 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.273275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.276065 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.276079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.277785 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.277792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.278080 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.278155 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:33:16.278278 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:33:16.279432 (+ 1154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7872,"mutex_wait_us":1107,"thread_start_us":621,"threads_started":1}
I20260926 07:33:16.283601 24379 tablet_service.cc:1467] Processing CreateTablet for tablet 4f99f1d37936423f983c3cfcc12fb414 (DEFAULT_TABLE table=test-table [id=c90fcc7ee9974572a8ad2cde173293cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:16.283601 24246 tablet_service.cc:1467] Processing CreateTablet for tablet 4f99f1d37936423f983c3cfcc12fb414 (DEFAULT_TABLE table=test-table [id=c90fcc7ee9974572a8ad2cde173293cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:16.285130 24379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f1d37936423f983c3cfcc12fb414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:16.285140 24246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f1d37936423f983c3cfcc12fb414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:16.284791 24112 tablet_service.cc:1467] Processing CreateTablet for tablet 4f99f1d37936423f983c3cfcc12fb414 (DEFAULT_TABLE table=test-table [id=c90fcc7ee9974572a8ad2cde173293cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:16.285797 24112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f1d37936423f983c3cfcc12fb414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:16.296885 24462 tablet_bootstrap.cc:492] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: Bootstrap starting.
I20260926 07:33:16.299857 24464 tablet_bootstrap.cc:492] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: Bootstrap starting.
I20260926 07:33:16.300525 24462 tablet_bootstrap.cc:654] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:16.302346 24462 log.cc:824] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:16.303522 24464 tablet_bootstrap.cc:654] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:16.303877 24463 tablet_bootstrap.cc:492] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: Bootstrap starting.
I20260926 07:33:16.304804 24464 log.cc:824] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:16.307372 24464 tablet_bootstrap.cc:492] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: No bootstrap required, opened a new log
I20260926 07:33:16.307444 24462 tablet_bootstrap.cc:492] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: No bootstrap required, opened a new log
I20260926 07:33:16.307471 24463 tablet_bootstrap.cc:654] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:16.307667 24464 ts_tablet_manager.cc:1397] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:33:16.307719 24462 ts_tablet_manager.cc:1397] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 07:33:16.308706 24463 log.cc:824] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:16.311748 24463 tablet_bootstrap.cc:492] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: No bootstrap required, opened a new log
I20260926 07:33:16.312017 24463 ts_tablet_manager.cc:1397] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260926 07:33:16.315928 24464 raft_consensus.cc:348] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.316319 24464 raft_consensus.cc:374] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:16.316452 24464 raft_consensus.cc:729] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f689034f9dbf4c38b04dbf6be044f541, State: Initialized, Role: FOLLOWER
I20260926 07:33:16.317169 24464 consensus_queue.cc:260] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.318138 24462 raft_consensus.cc:348] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.318553 24462 raft_consensus.cc:374] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:16.318656 24462 raft_consensus.cc:729] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbf82067b104668a51f3f195295b5a3, State: Initialized, Role: FOLLOWER
I20260926 07:33:16.319278 24462 consensus_queue.cc:260] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.322342 24462 ts_tablet_manager.cc:1428] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 07:33:16.322443 24463 raft_consensus.cc:348] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.323298 24463 raft_consensus.cc:374] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:16.323406 24444 heartbeater.cc:503] Master 127.22.221.126:43967 was elected leader, sending a full tablet report...
I20260926 07:33:16.323417 24463 raft_consensus.cc:729] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd747871a9d24aeeb174958d558416d5, State: Initialized, Role: FOLLOWER
I20260926 07:33:16.323976 24464 ts_tablet_manager.cc:1428] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 07:33:16.324115 24463 consensus_queue.cc:260] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.325277 24311 heartbeater.cc:503] Master 127.22.221.126:43967 was elected leader, sending a full tablet report...
I20260926 07:33:16.325927 24463 ts_tablet_manager.cc:1428] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 07:33:16.347882 24469 raft_consensus.cc:482] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:16.348204 24469 raft_consensus.cc:504] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.349841 24469 leader_election.cc:288] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd747871a9d24aeeb174958d558416d5 (127.22.221.66:45803), f689034f9dbf4c38b04dbf6be044f541 (127.22.221.67:45331)
I20260926 07:33:16.356842 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.350428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:39093 (local address 127.22.221.67:45331)
0926 07:33:16.350848 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.350854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.351051 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:33:16.351292 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.351298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.351378 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.351476 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:16.352123 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.352936 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.354956 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.355866 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.356000 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.356053 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.356450 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.356580 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260926 07:33:16.356983 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.350091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:45331 (local address 127.22.221.65:39093)
0926 07:33:16.350902 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:16.350946 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:16.351105 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:16.351511 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:16.351516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:16.351539 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:16.351962 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.351977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.353076 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.353082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.354792 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.354803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.356070 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.356079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.356299 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.356322 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:33:16.356400 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:33:16.356509 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":6,"thread_start_us":425,"threads_started":1}
I20260926 07:33:16.357398 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.350256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:45967 (local address 127.22.221.66:45803)
0926 07:33:16.350660 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.350667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.351304 (+ 637us) server_negotiation.cc:407] Connection header received
0926 07:33:16.351485 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.351490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.351565 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.351666 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:16.352253 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.352942 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.355095 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.355999 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.356152 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.356220 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.356673 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.356804 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260926 07:33:16.357409 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.350233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:45803 (local address 127.22.221.65:45967)
0926 07:33:16.351138 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:16.351151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:16.351332 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:16.351691 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:16.351694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:16.351714 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:16.352099 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.352110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.353080 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.353086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.354947 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:16.354960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.356212 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.356237 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:16.356494 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.356528 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:33:16.356618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:33:16.356790 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":45,"thread_start_us":572,"threads_started":1}
I20260926 07:33:16.358065 24399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f1d37936423f983c3cfcc12fb414" candidate_uuid: "3fbf82067b104668a51f3f195295b5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f689034f9dbf4c38b04dbf6be044f541" is_pre_election: true
I20260926 07:33:16.358337 24266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f1d37936423f983c3cfcc12fb414" candidate_uuid: "3fbf82067b104668a51f3f195295b5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd747871a9d24aeeb174958d558416d5" is_pre_election: true
I20260926 07:33:16.358552 24399 raft_consensus.cc:2454] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fbf82067b104668a51f3f195295b5a3 in term 0.
I20260926 07:33:16.358820 24266 raft_consensus.cc:2454] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fbf82067b104668a51f3f195295b5a3 in term 0.
I20260926 07:33:16.359292 24066 leader_election.cc:302] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fbf82067b104668a51f3f195295b5a3, f689034f9dbf4c38b04dbf6be044f541; no voters:
I20260926 07:33:16.359704 24469 raft_consensus.cc:2781] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:16.359849 24469 raft_consensus.cc:482] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:16.359956 24469 raft_consensus.cc:3037] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:16.362473 24469 raft_consensus.cc:504] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.363055 24469 leader_election.cc:288] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [CANDIDATE]: Term 1 election: Requested vote from peers bd747871a9d24aeeb174958d558416d5 (127.22.221.66:45803), f689034f9dbf4c38b04dbf6be044f541 (127.22.221.67:45331)
I20260926 07:33:16.363533 24266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f1d37936423f983c3cfcc12fb414" candidate_uuid: "3fbf82067b104668a51f3f195295b5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd747871a9d24aeeb174958d558416d5"
I20260926 07:33:16.363577 24399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f1d37936423f983c3cfcc12fb414" candidate_uuid: "3fbf82067b104668a51f3f195295b5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f689034f9dbf4c38b04dbf6be044f541"
I20260926 07:33:16.363719 24266 raft_consensus.cc:3037] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:16.363798 24399 raft_consensus.cc:3037] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:16.365959 24399 raft_consensus.cc:2454] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fbf82067b104668a51f3f195295b5a3 in term 1.
I20260926 07:33:16.365962 24266 raft_consensus.cc:2454] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fbf82067b104668a51f3f195295b5a3 in term 1.
I20260926 07:33:16.366365 24067 leader_election.cc:302] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fbf82067b104668a51f3f195295b5a3, bd747871a9d24aeeb174958d558416d5; no voters:
I20260926 07:33:16.366662 24469 raft_consensus.cc:2781] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:16.367133 24469 raft_consensus.cc:686] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fbf82067b104668a51f3f195295b5a3, State: Running, Role: LEADER
I20260926 07:33:16.367658 24469 consensus_queue.cc:237] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } }
I20260926 07:33:16.371918 23988 catalog_manager.cc:5495] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fbf82067b104668a51f3f195295b5a3 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "3fbf82067b104668a51f3f195295b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd747871a9d24aeeb174958d558416d5" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 45803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f689034f9dbf4c38b04dbf6be044f541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fbf82067b104668a51f3f195295b5a3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40119 } health_report { overall_health: HEALTHY } } }
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:16.402943 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.388425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.22.221.65:40119)
0926 07:33:16.388669 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.388677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.389065 (+ 388us) server_negotiation.cc:407] Connection header received
0926 07:33:16.389263 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.389270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.389338 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.389442 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.391003 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.391778 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.393049 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.393479 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.393622 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.397747 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.397759 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.397775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.397813 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.400511 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.401127 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.401133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.401136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.401295 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.401619 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.401622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.401624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.401946 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.402161 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.402385 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.402584 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260926 07:33:16.433543 24445 tablet.cc:2369] T 4f99f1d37936423f983c3cfcc12fb414 P f689034f9dbf4c38b04dbf6be044f541: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:16.450646 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.435850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.22.221.65:40119)
0926 07:33:16.436123 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.436131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.436491 (+ 360us) server_negotiation.cc:407] Connection header received
0926 07:33:16.436701 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.436711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.436795 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.436903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.438391 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.439184 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.440582 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.441002 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.441138 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.445226 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.445231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.445234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.445318 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.447883 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.448478 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.448482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.448484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.448614 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.449056 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.449060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.449062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.449525 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.449840 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.450139 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.450402 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260926 07:33:16.451613 24312 tablet.cc:2369] T 4f99f1d37936423f983c3cfcc12fb414 P bd747871a9d24aeeb174958d558416d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:33:16.461087 24178 tablet.cc:2369] T 4f99f1d37936423f983c3cfcc12fb414 P 3fbf82067b104668a51f3f195295b5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:16.501160 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.484168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.22.221.65:40119)
0926 07:33:16.484436 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.484443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.484678 (+ 235us) server_negotiation.cc:407] Connection header received
0926 07:33:16.484825 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.484829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.484909 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.485011 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.486299 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.487037 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.488267 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.488678 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.488774 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.496162 (+ 7388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.496167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.496170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.496208 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.498846 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.499507 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.499511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.499513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.499641 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.499968 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.499971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.499973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.500219 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.500415 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.500610 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.500773 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 07:33:16.502936 24092 tablet_service.cc:2908] Rejecting scan request for tablet 4f99f1d37936423f983c3cfcc12fb414: Uninitialized: clean time has not yet been initialized
I20260926 07:33:16.506150 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24049
I20260926 07:33:16.518846 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.507300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.22.221.65:40119)
0926 07:33:16.507462 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.507466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.507796 (+ 330us) server_negotiation.cc:407] Connection header received
0926 07:33:16.507925 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.507928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.507986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.508068 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.509224 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.509940 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.511095 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.511669 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.511820 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.513992 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.513997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.514000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.514030 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.516705 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.517296 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.517299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.517301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.517422 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.517714 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.517717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.517719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.517974 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.518155 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.518400 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.518578 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:33:16.519537 24172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:16.611606 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24049
I20260926 07:33:16.630590 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24183
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:16.651532 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.632124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.22.221.66:45803)
0926 07:33:16.632381 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.632387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.632473 (+ 86us) server_negotiation.cc:407] Connection header received
0926 07:33:16.632760 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.632765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.632854 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.632966 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.634615 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.636651 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.638699 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.639462 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.639619 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.644942 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.644966 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.644982 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.645016 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.648513 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.649195 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.649205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.649209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.649344 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.649730 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.649735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.649738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.650281 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.650611 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.650904 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.651117 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 07:33:16.652235 24306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:16.746580 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24183
I20260926 07:33:16.763512 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24316
Sep 26 07:33:16 dist-test-slave-qlq8 krb5kdc[23943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:16.784358 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:16.764312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.22.221.67:45331)
0926 07:33:16.764503 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:16.764507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:16.764544 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:33:16.764694 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:16.764698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:16.764761 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:16.764853 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:16.766092 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.766805 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.768158 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:16.768546 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:16.768639 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:16.778956 (+ 10317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:16.778974 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:16.778991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:16.779030 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:16.781894 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.782531 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.782538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.782540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.782666 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:16.783013 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:16.783016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:16.783017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:16.783453 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:16.783661 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:16.783943 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 07:33:16.784104 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:33:16.785019 24439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:16.872349 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24316
I20260926 07:33:16.888862 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 23956
I20260926 07:33:16.889990 24018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:16.998097 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 23956
2026-09-26T07:33:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4056 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:17 dist-test-slave-qlq8 krb5kdc[24492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:17 dist-test-slave-qlq8 krb5kdc[24492](info): set up 2 sockets
Sep 26 07:33:17 dist-test-slave-qlq8 krb5kdc[24492](info): commencing operation
krb5kdc: starting...
W20260926 07:33:18.087361 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:18 dist-test-slave-qlq8 krb5kdc[24492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:18.275295 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37189
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:18.605054 24505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:18.605352 24505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:18.605513 24505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:18.613435 24505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:18.613551 24505 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:33:18.613624 24505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:18.613744 24505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:18.613824 24505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:18.613936 24505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:18.624110 24505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37189
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:18.625468 24505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:18.626883 24505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:18.639470 24510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:18.639416 24511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:18.639712 24505 server_base.cc:1034] running on GCE node
W20260926 07:33:18.639470 24513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:18.640919 24505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:18.642433 24505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:18.643642 24505 hybrid_clock.cc:648] HybridClock initialized: now 1790407998643556 us; error 65 us; skew 500 ppm
I20260926 07:33:18.644001 24505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:18 dist-test-slave-qlq8 krb5kdc[24492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407998, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:18.647097 24505 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:18.648694 24505 webserver.cc:460] Webserver started at http://127.22.221.126:44467/ using document root <none> and password file <none>
I20260926 07:33:18.649381 24505 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:18.649483 24505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:18.649753 24505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:18.652778 24505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "693036aa441c4925b602bc23881a8fbe"
format_stamp: "Formatted at 2026-09-26 07:33:18 on dist-test-slave-qlq8"
server_key: "705c8c1b5ff443cc3d26c46cfac130b3"
server_key_iv: "432ac770c671d20f2d6c246947eead38"
server_key_version: "encryptionkey@0"
I20260926 07:33:18.653551 24505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "693036aa441c4925b602bc23881a8fbe"
format_stamp: "Formatted at 2026-09-26 07:33:18 on dist-test-slave-qlq8"
server_key: "705c8c1b5ff443cc3d26c46cfac130b3"
server_key_iv: "432ac770c671d20f2d6c246947eead38"
server_key_version: "encryptionkey@0"
I20260926 07:33:18.658764 24505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:18.662257 24520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:18.663713 24505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:18.663898 24505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "693036aa441c4925b602bc23881a8fbe"
format_stamp: "Formatted at 2026-09-26 07:33:18 on dist-test-slave-qlq8"
server_key: "705c8c1b5ff443cc3d26c46cfac130b3"
server_key_iv: "432ac770c671d20f2d6c246947eead38"
server_key_version: "encryptionkey@0"
I20260926 07:33:18.664126 24505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:18.704073 24505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:18.708386 24505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:18.708705 24505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:18.731031 24505 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37189
I20260926 07:33:18.731071 24581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37189 every 8 connection(s)
I20260926 07:33:18.732502 24505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:18.736516 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24505
I20260926 07:33:18.736542 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:18.736757 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:18.737425 23413 external_mini_cluster.cc:1428] Setting key 5a76a63175de69e6170cee46d0eb1a99
I20260926 07:33:18.751597 24582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: Bootstrap starting.
Sep 26 07:33:18 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:18.755029 24582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:18.756582 24582 log.cc:824] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:18.760186 24582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: No bootstrap required, opened a new log
I20260926 07:33:18.763346 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:18.739578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.22.221.126:37189)
0926 07:33:18.740473 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:18.740488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:18.740546 (+ 58us) server_negotiation.cc:407] Connection header received
0926 07:33:18.743488 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:18.743521 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:18.744087 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:18.744586 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:18.745879 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:18.747933 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:18.749207 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:18.750280 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:18.750475 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:18.754515 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:18.754525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:18.754545 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:18.754598 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:18.758448 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:18.759264 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:18.759274 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:18.759282 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:18.759495 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:18.759870 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:18.759875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:18.759878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:18.760629 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:18.760955 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:18.761348 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 07:33:18.761824 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":380,"threads_started":1}
I20260926 07:33:18.770036 24582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693036aa441c4925b602bc23881a8fbe" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37189 } }
I20260926 07:33:18.770421 24582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:18.770495 24582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693036aa441c4925b602bc23881a8fbe, State: Initialized, Role: FOLLOWER
I20260926 07:33:18.771088 24582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693036aa441c4925b602bc23881a8fbe" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37189 } }
I20260926 07:33:18.771428 24582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:18.771581 24582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:18.771806 24582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:18.774168 24582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693036aa441c4925b602bc23881a8fbe" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37189 } }
I20260926 07:33:18.774755 24582 leader_election.cc:302] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693036aa441c4925b602bc23881a8fbe; no voters:
I20260926 07:33:18.775481 24582 leader_election.cc:288] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:18.775640 24587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:18.777341 24587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [term 1 LEADER]: Becoming Leader. State: Replica: 693036aa441c4925b602bc23881a8fbe, State: Running, Role: LEADER
I20260926 07:33:18.777841 24582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:18.777861 24587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693036aa441c4925b602bc23881a8fbe" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37189 } }
I20260926 07:33:18.781075 24588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "693036aa441c4925b602bc23881a8fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693036aa441c4925b602bc23881a8fbe" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37189 } } }
I20260926 07:33:18.781031 24589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693036aa441c4925b602bc23881a8fbe. Latest consensus state: current_term: 1 leader_uuid: "693036aa441c4925b602bc23881a8fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693036aa441c4925b602bc23881a8fbe" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37189 } } }
I20260926 07:33:18.781493 24589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:18.781497 24588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:18.783363 24596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:18.790402 24596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:18.800854 24596 catalog_manager.cc:1347] Generated new cluster ID: 50d0e7390feb45e38ca341cfa9f1fb1f
I20260926 07:33:18.800989 24596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:18.829875 24596 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:18.831867 24596 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:18.846316 24596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: Generated new TSK 0
I20260926 07:33:18.847447 24596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:18.904533 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37189
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:19.236227 24608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:19.236558 24608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:19.236737 24608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:19.244149 24608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:19.244299 24608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:19.244463 24608 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:33:19.244607 24608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:19.254489 24608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:37189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:19.255808 24608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:19.257134 24608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:19.270428 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:19.270469 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:19.270496 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:19.271695 24608 server_base.cc:1034] running on GCE node
I20260926 07:33:19.272362 24608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:19.273370 24608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:19.274576 24608 hybrid_clock.cc:648] HybridClock initialized: now 1790407999274497 us; error 56 us; skew 500 ppm
I20260926 07:33:19.274832 24608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:19 dist-test-slave-qlq8 krb5kdc[24492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:19.278074 24608 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:19.279574 24608 webserver.cc:460] Webserver started at http://127.22.221.65:39867/ using document root <none> and password file <none>
I20260926 07:33:19.280270 24608 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:19.280357 24608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:19.280622 24608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:19.283921 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "52f9bc7c57bf40efa28de91686331db8"
format_stamp: "Formatted at 2026-09-26 07:33:19 on dist-test-slave-qlq8"
server_key: "927f47b81d3721f67fcb8abd8dfac77b"
server_key_iv: "ac52d8270fba1f39b98052009b389053"
server_key_version: "encryptionkey@0"
I20260926 07:33:19.284685 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "52f9bc7c57bf40efa28de91686331db8"
format_stamp: "Formatted at 2026-09-26 07:33:19 on dist-test-slave-qlq8"
server_key: "927f47b81d3721f67fcb8abd8dfac77b"
server_key_iv: "ac52d8270fba1f39b98052009b389053"
server_key_version: "encryptionkey@0"
I20260926 07:33:19.290051 24608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 07:33:19.293546 24623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:19.294894 24608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:19.295091 24608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "52f9bc7c57bf40efa28de91686331db8"
format_stamp: "Formatted at 2026-09-26 07:33:19 on dist-test-slave-qlq8"
server_key: "927f47b81d3721f67fcb8abd8dfac77b"
server_key_iv: "ac52d8270fba1f39b98052009b389053"
server_key_version: "encryptionkey@0"
I20260926 07:33:19.295356 24608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:19.309599 24608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:19.313973 24608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:19.314248 24608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:19.316799 24608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:19.316922 24608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:19.317057 24608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:19.317130 24608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:33:19 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:19.342095 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.317610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:58149 (local address 127.22.221.126:37189)
0926 07:33:19.318103 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:19.318108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:19.319272 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 07:33:19.321254 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:19.321260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:19.321348 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:19.321487 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:19.323840 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.324917 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.327625 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.328209 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.328351 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.333939 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:19.333948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:19.333956 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:19.334010 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:19.338056 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.339104 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.339109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.339113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.339312 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.340036 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.340042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.340047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.340354 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:19.340510 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:19.341451 (+ 941us) server_negotiation.cc:299] Negotiation successful
0926 07:33:19.341677 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":294,"threads_started":1}
I20260926 07:33:19.343722 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.318293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.22.221.65:58149)
0926 07:33:19.319097 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:19.319133 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:19.320976 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:19.321756 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:19.321769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:19.322624 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:19.323651 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.323672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.325078 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.325085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.327458 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.327475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.328506 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.328514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.328733 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.329623 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:19.329648 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:19.333578 (+ 3930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:19.338226 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.338238 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.338260 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.338971 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.339453 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.339461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.339473 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.339870 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.340485 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:19.340493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:19.340996 (+ 503us) client_negotiation.cc:769] Sending connection context
0926 07:33:19.341365 (+ 369us) client_negotiation.cc:240] Negotiation successful
0926 07:33:19.341987 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260926 07:33:19.362224 24608 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:36795
I20260926 07:33:19.362242 24743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:36795 every 8 connection(s)
I20260926 07:33:19.363618 24608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:19.364491 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24608
I20260926 07:33:19.364732 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:19.365442 23413 external_mini_cluster.cc:1428] Setting key b8556d92371d0bdc55e1a097a7d0ed51
I20260926 07:33:19.366055 24744 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37189
I20260926 07:33:19.366338 24744 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:19.367373 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.349688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:49297 (local address 127.22.221.126:37189)
0926 07:33:19.349917 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:19.349925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:19.350555 (+ 630us) server_negotiation.cc:407] Connection header received
0926 07:33:19.350754 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:19.350758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:19.350818 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:19.350895 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:19.352625 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.353347 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.355914 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.356451 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.356593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.359362 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:19.359371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:19.359378 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:19.359434 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:19.362276 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.363007 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.363014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.363018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.363156 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.363546 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.363551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.363555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.363907 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:19.364069 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:19.366643 (+ 2574us) server_negotiation.cc:299] Negotiation successful
0926 07:33:19.366876 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:33:19.367415 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.349676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.22.221.65:49297)
0926 07:33:19.350431 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:19.350457 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:19.350634 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:19.350974 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:19.350982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:19.351802 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:19.352444 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.352466 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.353487 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.353495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.355733 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.355760 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.356887 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.356892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.357142 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.357910 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:19.357940 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:19.359080 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:19.362472 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.362478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.362482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.362887 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.363304 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.363308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.363310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.363436 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.364023 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:19.364027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:19.366303 (+ 2276us) client_negotiation.cc:769] Sending connection context
0926 07:33:19.366507 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 07:33:19.366871 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":368,"threads_started":1}
I20260926 07:33:19.367563 24744 heartbeater.cc:511] Master 127.22.221.126:37189 requested a full tablet report, sending...
I20260926 07:33:19.370455 24537 ts_manager.cc:194] Registered new tserver with Master: 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795)
I20260926 07:33:19.372138 24537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:58149
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:19.406646 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37189
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:19.726418 24748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:19.726694 24748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:19.726868 24748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:19.734145 24748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:19.734289 24748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:19.734447 24748 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:33:19.734584 24748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:19.744269 24748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:37189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:19.745622 24748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:19.747076 24748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:19.759923 24753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:19.760004 24754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:19.760223 24756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:19.761019 24748 server_base.cc:1034] running on GCE node
I20260926 07:33:19.761663 24748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:19.762710 24748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:19.763962 24748 hybrid_clock.cc:648] HybridClock initialized: now 1790407999763899 us; error 63 us; skew 500 ppm
I20260926 07:33:19.764278 24748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:19 dist-test-slave-qlq8 krb5kdc[24492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:19.767553 24748 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:19.769213 24748 webserver.cc:460] Webserver started at http://127.22.221.66:33607/ using document root <none> and password file <none>
I20260926 07:33:19.769923 24748 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:19.770011 24748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:19.770275 24748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:19.773370 24748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "1c9627179fdc492fbb826ddd2e9824bb"
format_stamp: "Formatted at 2026-09-26 07:33:19 on dist-test-slave-qlq8"
server_key: "b705f60ee16e3dce651e236238f5677f"
server_key_iv: "6dfc1ea30fd4c3371cf14b2c5b53a287"
server_key_version: "encryptionkey@0"
I20260926 07:33:19.774160 24748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "1c9627179fdc492fbb826ddd2e9824bb"
format_stamp: "Formatted at 2026-09-26 07:33:19 on dist-test-slave-qlq8"
server_key: "b705f60ee16e3dce651e236238f5677f"
server_key_iv: "6dfc1ea30fd4c3371cf14b2c5b53a287"
server_key_version: "encryptionkey@0"
I20260926 07:33:19.779812 24748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:33:19.783309 24763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:19.784647 24748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:19.784824 24748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "1c9627179fdc492fbb826ddd2e9824bb"
format_stamp: "Formatted at 2026-09-26 07:33:19 on dist-test-slave-qlq8"
server_key: "b705f60ee16e3dce651e236238f5677f"
server_key_iv: "6dfc1ea30fd4c3371cf14b2c5b53a287"
server_key_version: "encryptionkey@0"
I20260926 07:33:19.785051 24748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:19.811419 24748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:19.815517 24748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:19.815806 24748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:19.818408 24748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:19.818545 24748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:19.818707 24748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:19.818786 24748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:33:19 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:19.846613 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.819349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:47035 (local address 127.22.221.126:37189)
0926 07:33:19.819541 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:19.819548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:19.821064 (+ 1516us) server_negotiation.cc:407] Connection header received
0926 07:33:19.824277 (+ 3213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:19.824283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:19.824391 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:19.824492 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:19.827568 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.828580 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.831940 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.832530 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.832657 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.838524 (+ 5867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:19.838534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:19.838541 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:19.838596 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:19.842248 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.843575 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.843584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.843589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.843769 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.844436 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.844439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.844441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.844718 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:19.844879 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:19.845874 (+ 995us) server_negotiation.cc:299] Negotiation successful
0926 07:33:19.846194 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:33:19.848088 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.819935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.22.221.66:47035)
0926 07:33:19.820852 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:19.820909 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:19.823834 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:19.825165 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:19.825180 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:19.826203 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:19.827328 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.827361 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.828792 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.828801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.831748 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.831770 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.833002 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.833036 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.833337 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.834145 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:19.834191 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:19.838205 (+ 4014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:19.842426 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.842438 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.842462 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.843367 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.843914 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.843920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.843925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.844243 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.844868 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:19.844876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:19.845303 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 07:33:19.845727 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 07:33:19.846420 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":346,"threads_started":1}
I20260926 07:33:19.862807 24748 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:43235
I20260926 07:33:19.862891 24882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:43235 every 8 connection(s)
I20260926 07:33:19.864416 24748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:19.866731 24883 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37189
I20260926 07:33:19.867076 24883 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:19.868096 24883 heartbeater.cc:511] Master 127.22.221.126:37189 requested a full tablet report, sending...
I20260926 07:33:19.869541 24537 ts_manager.cc:194] Registered new tserver with Master: 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235)
I20260926 07:33:19.870630 24537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:47035
I20260926 07:33:19.870814 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.854321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:48565 (local address 127.22.221.126:37189)
0926 07:33:19.854515 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:19.854522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:19.855552 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 07:33:19.855792 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:19.855800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:19.855887 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:19.856008 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:19.857834 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.858598 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.861089 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.861656 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.861783 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.864273 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:19.864278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:19.864281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:19.864312 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:19.868056 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.868865 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.868869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.868871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.869046 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:19.869455 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:19.869459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:19.869461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:19.869826 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:19.869944 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:19.870387 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 07:33:19.870547 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:33:19.870990 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:19.854400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.22.221.66:48565)
0926 07:33:19.855436 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:19.855455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:19.855636 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:19.856225 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:19.856233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:19.857029 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:19.857652 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.857673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.858763 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.858776 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.860955 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:19.860971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:19.861885 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:19.861894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:19.862113 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:19.862965 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:19.862989 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:19.864100 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:19.868193 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.868205 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.868209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.868669 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.869179 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:19.869183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:19.869186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:19.869342 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:19.869948 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:19.869956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:19.870134 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 07:33:19.870293 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 07:33:19.870579 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":724,"threads_started":1}
I20260926 07:33:19.874203 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24748
I20260926 07:33:19.874457 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:19.874887 23413 external_mini_cluster.cc:1428] Setting key 9d2fdc24cb4417e44f34094812df4d55
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:19.915498 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37189
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:20.235536 24887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:20.235802 24887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:20.235915 24887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:20.242969 24887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:20.243072 24887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:20.243153 24887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:33:20.243249 24887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:20.252952 24887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:37189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:20.254165 24887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:20.255523 24887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:20.268224 24892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:20.268301 24893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:20.268363 24895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:20.269443 24887 server_base.cc:1034] running on GCE node
I20260926 07:33:20.270040 24887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:20.270889 24887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:20.272122 24887 hybrid_clock.cc:648] HybridClock initialized: now 1790408000272052 us; error 48 us; skew 500 ppm
I20260926 07:33:20.272429 24887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:20 dist-test-slave-qlq8 krb5kdc[24492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:20.275408 24887 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:20.276800 24887 webserver.cc:460] Webserver started at http://127.22.221.67:44135/ using document root <none> and password file <none>
I20260926 07:33:20.277489 24887 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:20.277616 24887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:20.277932 24887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:20.280797 24887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "c68c3391ff6743a88abfef8aa4ec866a"
format_stamp: "Formatted at 2026-09-26 07:33:20 on dist-test-slave-qlq8"
server_key: "77d57f18871b63a07318694a890112a8"
server_key_iv: "b024cd631b9aeba43da21e164814ecca"
server_key_version: "encryptionkey@0"
I20260926 07:33:20.281612 24887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "c68c3391ff6743a88abfef8aa4ec866a"
format_stamp: "Formatted at 2026-09-26 07:33:20 on dist-test-slave-qlq8"
server_key: "77d57f18871b63a07318694a890112a8"
server_key_iv: "b024cd631b9aeba43da21e164814ecca"
server_key_version: "encryptionkey@0"
I20260926 07:33:20.286584 24887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:20.290032 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:20.291388 24887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:20.291568 24887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "c68c3391ff6743a88abfef8aa4ec866a"
format_stamp: "Formatted at 2026-09-26 07:33:20 on dist-test-slave-qlq8"
server_key: "77d57f18871b63a07318694a890112a8"
server_key_iv: "b024cd631b9aeba43da21e164814ecca"
server_key_version: "encryptionkey@0"
I20260926 07:33:20.291743 24887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:20.310629 24887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:20.314553 24887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:20.314826 24887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:20.317574 24887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:20.317684 24887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:20.317893 24887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:20.317967 24887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:33:20 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:20.340862 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.318096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:58499 (local address 127.22.221.126:37189)
0926 07:33:20.318303 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.318311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.319622 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 07:33:20.321729 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.321738 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.321834 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.321961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:20.324244 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.325570 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.328167 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.328585 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.328753 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.333413 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:20.333418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:20.333421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:20.333454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:20.336583 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.337744 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.337756 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.337758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.337978 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.338620 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.338628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.338634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.339062 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:20.339260 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.340267 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.340515 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:33:20.341883 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.318639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.22.221.67:58499)
0926 07:33:20.319466 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.319508 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.321353 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.322255 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.322277 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.323097 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:20.324005 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.324029 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.325761 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.325769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.328001 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.328013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.328859 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.328869 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.329240 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.330177 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:20.330216 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:20.333162 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:20.336745 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:20.336761 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:20.336784 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:20.337591 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:20.338131 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:20.338136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:20.338140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:20.338459 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:20.339262 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:20.339273 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:20.339750 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.340112 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.340650 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":331,"threads_started":1}
I20260926 07:33:20.365787 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.349714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.22.221.67:60145)
0926 07:33:20.350406 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.350435 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.350601 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.350987 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.350990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.351486 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:20.351758 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.351766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.353542 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.353546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.354527 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.354539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.355790 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.355801 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.356020 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.356685 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:20.356714 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:20.358025 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:20.362621 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:20.362630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:20.362637 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:20.363295 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:20.363740 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:20.363744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:20.363747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:20.363900 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:20.364626 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:20.364631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:20.364833 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.364992 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.365322 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":394,"threads_started":1}
I20260926 07:33:20.366056 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.349199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:60145 (local address 127.22.221.126:37189)
0926 07:33:20.349426 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.349432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.350537 (+ 1105us) server_negotiation.cc:407] Connection header received
0926 07:33:20.350739 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.350747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.350844 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.350971 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:20.351895 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.353309 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.354662 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.355555 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.355715 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.358244 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:20.358254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:20.358262 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:20.358311 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:20.362436 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.363425 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.363430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.363435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.363605 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.364022 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.364030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.364036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.364483 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:20.364672 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.365143 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.365451 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:33:20.374809 24744 heartbeater.cc:503] Master 127.22.221.126:37189 was elected leader, sending a full tablet report...
I20260926 07:33:20.380254 24887 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:40907
I20260926 07:33:20.380297 25021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:40907 every 8 connection(s)
I20260926 07:33:20.381450 24887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:20.383291 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 24887
I20260926 07:33:20.383491 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:20.383719 25022 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37189
I20260926 07:33:20.383950 25022 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:20.384120 23413 external_mini_cluster.cc:1428] Setting key 5dff5532ad31498a59324360a32b3882
I20260926 07:33:20.384752 25022 heartbeater.cc:511] Master 127.22.221.126:37189 requested a full tablet report, sending...
I20260926 07:33:20.386075 24537 ts_manager.cc:194] Registered new tserver with Master: c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907)
I20260926 07:33:20.387005 24537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:58499
I20260926 07:33:20.387384 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:33:20 dist-test-slave-qlq8 krb5kdc[24492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:33:20 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:20.432560 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.419156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.22.221.126:37189)
0926 07:33:20.419363 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.419368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.419623 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:33:20.419769 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.419773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.419838 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.419943 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:20.421279 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.422005 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.423223 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.423712 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.423808 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.427660 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:20.427665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:20.427668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:20.427701 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:20.430289 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.430848 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.430851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.430854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.430989 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.431352 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.431356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.431358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.431620 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:20.431800 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.432054 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.432214 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:33:20.436757 24537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:20.443544 24537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:20.479211 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.455827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:36795 (local address 127.0.0.1:56668)
0926 07:33:20.461471 (+ 5644us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.461521 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.461730 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.470228 (+ 8498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.470234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.470260 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.470607 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.470620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.474446 (+ 3826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.474451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.476469 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.476481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.478220 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.478229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.478432 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.478490 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.478591 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.478734 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5295,"mutex_wait_us":52,"thread_start_us":5085,"threads_started":1}
I20260926 07:33:20.480082 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.456735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.22.221.67:40907)
0926 07:33:20.468558 (+ 11823us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.468567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.471159 (+ 2592us) server_negotiation.cc:407] Connection header received
0926 07:33:20.471480 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.471488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.471764 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.471939 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.473036 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.474645 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.477446 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.478851 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.478987 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.479264 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.479562 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.479744 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11654,"thread_start_us":11826,"threads_started":1}
I20260926 07:33:20.480125 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.455964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.22.221.65:36795)
0926 07:33:20.469323 (+ 13359us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.469331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.469362 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:33:20.469470 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.469480 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.469766 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.470007 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.470798 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.474296 (+ 3498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.476638 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.477966 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.479381 (+ 1415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.479594 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.479665 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.479827 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13193,"thread_start_us":12955,"threads_started":1}
I20260926 07:33:20.480166 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.462100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43235 (local address 127.0.0.1:45806)
0926 07:33:20.471250 (+ 9150us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.471269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.471388 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.472492 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.472497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.472526 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.472998 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.473010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.475386 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.475392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.477547 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.477560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.479383 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.479389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.479558 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.479590 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.479668 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.479794 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9001,"mutex_wait_us":1,"thread_start_us":7518,"threads_started":1}
I20260926 07:33:20.480791 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.456427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:40907 (local address 127.0.0.1:35972)
0926 07:33:20.471030 (+ 14603us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.471054 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.471287 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.472300 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.472305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.472332 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.472795 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.472808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.474819 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.474828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.477238 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.477252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.479084 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.479091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.479323 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.479375 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.480295 (+ 920us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.480435 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13308,"mutex_wait_us":39,"thread_start_us":14517,"threads_started":1}
I20260926 07:33:20.481134 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.462136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.22.221.66:43235)
0926 07:33:20.468428 (+ 6292us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.468441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.471593 (+ 3152us) server_negotiation.cc:407] Connection header received
0926 07:33:20.471794 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.471803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.472065 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.472251 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.473217 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.475216 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.477714 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.479030 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.479149 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.480050 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.480124 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.480752 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6121,"thread_start_us":5990,"threads_started":1}
I20260926 07:33:20.484001 24674 tablet_service.cc:1467] Processing CreateTablet for tablet 48a0f59f71e54395adab1be417638009 (DEFAULT_TABLE table=test-table [id=9af891a8da874cb49682b6d2f0ceb6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:20.484366 24813 tablet_service.cc:1467] Processing CreateTablet for tablet 48a0f59f71e54395adab1be417638009 (DEFAULT_TABLE table=test-table [id=9af891a8da874cb49682b6d2f0ceb6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:20.484375 24952 tablet_service.cc:1467] Processing CreateTablet for tablet 48a0f59f71e54395adab1be417638009 (DEFAULT_TABLE table=test-table [id=9af891a8da874cb49682b6d2f0ceb6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:20.485512 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a0f59f71e54395adab1be417638009. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.485530 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a0f59f71e54395adab1be417638009. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.485644 24952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a0f59f71e54395adab1be417638009. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.497294 25041 tablet_bootstrap.cc:492] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Bootstrap starting.
I20260926 07:33:20.499715 25043 tablet_bootstrap.cc:492] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Bootstrap starting.
I20260926 07:33:20.500540 25042 tablet_bootstrap.cc:492] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Bootstrap starting.
I20260926 07:33:20.501983 25041 tablet_bootstrap.cc:654] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.503351 25041 log.cc:824] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:20.504168 25042 tablet_bootstrap.cc:654] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.505060 25043 tablet_bootstrap.cc:654] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.505319 25042 log.cc:824] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:20.506222 25043 log.cc:824] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:20.506573 25041 tablet_bootstrap.cc:492] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: No bootstrap required, opened a new log
I20260926 07:33:20.506870 25041 ts_tablet_manager.cc:1397] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 07:33:20.508373 25042 tablet_bootstrap.cc:492] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: No bootstrap required, opened a new log
I20260926 07:33:20.508625 25042 ts_tablet_manager.cc:1397] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:33:20.508769 25043 tablet_bootstrap.cc:492] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: No bootstrap required, opened a new log
I20260926 07:33:20.509016 25043 ts_tablet_manager.cc:1397] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260926 07:33:20.516157 25043 raft_consensus.cc:348] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.516552 25043 raft_consensus.cc:374] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.516673 25043 raft_consensus.cc:729] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9bc7c57bf40efa28de91686331db8, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.517338 25043 consensus_queue.cc:260] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.518232 25041 raft_consensus.cc:348] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.518702 25041 raft_consensus.cc:374] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.518848 25041 raft_consensus.cc:729] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9627179fdc492fbb826ddd2e9824bb, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.519157 25043 ts_tablet_manager.cc:1428] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260926 07:33:20.519042 25042 raft_consensus.cc:348] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.519486 25042 raft_consensus.cc:374] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.519614 25042 raft_consensus.cc:729] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68c3391ff6743a88abfef8aa4ec866a, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.519673 25041 consensus_queue.cc:260] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.520702 25042 consensus_queue.cc:260] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.521936 25022 heartbeater.cc:503] Master 127.22.221.126:37189 was elected leader, sending a full tablet report...
I20260926 07:33:20.522596 25042 ts_tablet_manager.cc:1428] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 07:33:20.523454 24883 heartbeater.cc:503] Master 127.22.221.126:37189 was elected leader, sending a full tablet report...
I20260926 07:33:20.524386 25041 ts_tablet_manager.cc:1428] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260926 07:33:20.535019 25047 raft_consensus.cc:482] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:20.535351 25047 raft_consensus.cc:504] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.536886 25047 leader_election.cc:288] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235), c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907)
I20260926 07:33:20.543640 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.537374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:50753 (local address 127.22.221.67:40907)
0926 07:33:20.537662 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.537668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.538157 (+ 489us) server_negotiation.cc:407] Connection header received
0926 07:33:20.538353 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.538358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.538443 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.538538 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.539141 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.539958 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.541926 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.542687 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.542809 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.542900 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.543224 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.543372 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 07:33:20.543677 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.537393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:59341 (local address 127.22.221.66:43235)
0926 07:33:20.537809 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.537817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.538361 (+ 544us) server_negotiation.cc:407] Connection header received
0926 07:33:20.538536 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.538540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.538609 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.538711 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.539412 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.540259 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.541998 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.542772 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.542941 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.543011 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.543324 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.543453 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260926 07:33:20.543753 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.537291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:40907 (local address 127.22.221.65:50753)
0926 07:33:20.538021 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.538054 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.538214 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.538562 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.538567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.538591 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.538995 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.539006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.540082 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.540087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.541773 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.541789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.542897 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.542904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.543071 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.543095 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.543168 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.543321 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":25,"thread_start_us":315,"threads_started":1}
I20260926 07:33:20.543926 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.537353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43235 (local address 127.22.221.65:59341)
0926 07:33:20.538238 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.538252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.538391 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.538736 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.538739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.538812 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.539211 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.539224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.540395 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.540399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.541817 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.541829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.543006 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.543011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.543118 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.543140 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.543279 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.543396 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":19,"spinlock_wait_cycles":24320,"thread_start_us":594,"threads_started":1}
I20260926 07:33:20.544783 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a0f59f71e54395adab1be417638009" candidate_uuid: "52f9bc7c57bf40efa28de91686331db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68c3391ff6743a88abfef8aa4ec866a" is_pre_election: true
I20260926 07:33:20.545051 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a0f59f71e54395adab1be417638009" candidate_uuid: "52f9bc7c57bf40efa28de91686331db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9627179fdc492fbb826ddd2e9824bb" is_pre_election: true
I20260926 07:33:20.545375 24976 raft_consensus.cc:2454] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f9bc7c57bf40efa28de91686331db8 in term 0.
I20260926 07:33:20.545539 24833 raft_consensus.cc:2454] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f9bc7c57bf40efa28de91686331db8 in term 0.
I20260926 07:33:20.546104 24626 leader_election.cc:302] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c9627179fdc492fbb826ddd2e9824bb, 52f9bc7c57bf40efa28de91686331db8, c68c3391ff6743a88abfef8aa4ec866a; no voters:
I20260926 07:33:20.546506 25047 raft_consensus.cc:2781] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:20.546653 25047 raft_consensus.cc:482] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:20.546765 25047 raft_consensus.cc:3037] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:20.549340 25047 raft_consensus.cc:504] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.549971 25047 leader_election.cc:288] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [CANDIDATE]: Term 1 election: Requested vote from peers 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235), c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907)
I20260926 07:33:20.550323 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a0f59f71e54395adab1be417638009" candidate_uuid: "52f9bc7c57bf40efa28de91686331db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9627179fdc492fbb826ddd2e9824bb"
I20260926 07:33:20.550522 24833 raft_consensus.cc:3037] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:20.550680 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a0f59f71e54395adab1be417638009" candidate_uuid: "52f9bc7c57bf40efa28de91686331db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68c3391ff6743a88abfef8aa4ec866a"
I20260926 07:33:20.550899 24976 raft_consensus.cc:3037] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:20.552870 24833 raft_consensus.cc:2454] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f9bc7c57bf40efa28de91686331db8 in term 1.
I20260926 07:33:20.553242 24625 leader_election.cc:302] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9627179fdc492fbb826ddd2e9824bb, 52f9bc7c57bf40efa28de91686331db8; no voters:
I20260926 07:33:20.553280 24976 raft_consensus.cc:2454] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f9bc7c57bf40efa28de91686331db8 in term 1.
I20260926 07:33:20.553594 25047 raft_consensus.cc:2781] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:20.554118 25047 raft_consensus.cc:686] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 1 LEADER]: Becoming Leader. State: Replica: 52f9bc7c57bf40efa28de91686331db8, State: Running, Role: LEADER
I20260926 07:33:20.554654 25047 consensus_queue.cc:237] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.559512 24536 catalog_manager.cc:5495] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "52f9bc7c57bf40efa28de91686331db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } health_report { overall_health: UNKNOWN } } }
I20260926 07:33:20.586335 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.580602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.22.221.65:36795)
0926 07:33:20.580831 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.580838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.580986 (+ 148us) server_negotiation.cc:407] Connection header received
0926 07:33:20.581311 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.581316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.581401 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.581570 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:20.581961 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.582738 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.584141 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.584556 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.584693 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.585191 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:20.585678 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:20.585869 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.585992 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.586075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:33:20.594110 24976 raft_consensus.cc:1261] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Refusing update from remote peer 52f9bc7c57bf40efa28de91686331db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:20.594110 24833 raft_consensus.cc:1261] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Refusing update from remote peer 52f9bc7c57bf40efa28de91686331db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:20.595108 25052 consensus_queue.cc:1035] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:20.595521 25047 consensus_queue.cc:1035] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:20.608345 25057 mvcc.cc:205] Tried to move back new op lower bound from 7333511170423554048 to 7333511170276282368. Current Snapshot: MvccSnapshot[applied={T|T < 7333511170423554048}]
I20260926 07:33:20.624606 24537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:20.624929 24537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:20.628623 24537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: Sending DeleteTablet for 3 replicas of tablet 48a0f59f71e54395adab1be417638009
I20260926 07:33:20.629671 24674 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a0f59f71e54395adab1be417638009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:20 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:56668
I20260926 07:33:20.629822 24813 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a0f59f71e54395adab1be417638009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:20 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:45806
I20260926 07:33:20.630492 24952 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a0f59f71e54395adab1be417638009 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:20 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:35972
I20260926 07:33:20.630645 25065 tablet_replica.cc:330] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: stopping tablet replica
I20260926 07:33:20.631088 25065 raft_consensus.cc:2229] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:20.631845 25065 raft_consensus.cc:2258] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:20.632040 24537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:20.632751 24537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:20.635739 25066 tablet_replica.cc:330] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: stopping tablet replica
I20260926 07:33:20.636230 25066 raft_consensus.cc:2229] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:20.636821 25066 raft_consensus.cc:2258] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:20.639873 25066 ts_tablet_manager.cc:1905] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:20.641922 24951 tablet_service.cc:1467] Processing CreateTablet for tablet 4d67bb7467734bfeb33a3a29f4d99063 (DEFAULT_TABLE table=test-table [id=bbf188160df14670a69fa3e536299f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:20.642551 24951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d67bb7467734bfeb33a3a29f4d99063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.642634 25065 ts_tablet_manager.cc:1905] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:20.645700 24674 tablet_service.cc:1467] Processing CreateTablet for tablet 4d67bb7467734bfeb33a3a29f4d99063 (DEFAULT_TABLE table=test-table [id=bbf188160df14670a69fa3e536299f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:20.646340 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d67bb7467734bfeb33a3a29f4d99063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.646526 24813 tablet_service.cc:1467] Processing CreateTablet for tablet 4d67bb7467734bfeb33a3a29f4d99063 (DEFAULT_TABLE table=test-table [id=bbf188160df14670a69fa3e536299f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:20.653127 25043 tablet_bootstrap.cc:492] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Bootstrap starting.
I20260926 07:33:20.653666 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d67bb7467734bfeb33a3a29f4d99063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.655567 25043 tablet_bootstrap.cc:654] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.660157 25041 tablet_bootstrap.cc:492] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Bootstrap starting.
I20260926 07:33:20.661890 25041 tablet_bootstrap.cc:654] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.668295 25042 tablet_bootstrap.cc:492] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Bootstrap starting.
I20260926 07:33:20.670300 25042 tablet_bootstrap.cc:654] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.671648 25065 ts_tablet_manager.cc:1918] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:20.672840 25065 log.cc:1196] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/48a0f59f71e54395adab1be417638009
I20260926 07:33:20.673869 25065 ts_tablet_manager.cc:1939] T 48a0f59f71e54395adab1be417638009 P 52f9bc7c57bf40efa28de91686331db8: Deleting consensus metadata
I20260926 07:33:20.679057 25042 tablet_bootstrap.cc:492] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: No bootstrap required, opened a new log
I20260926 07:33:20.679219 25042 ts_tablet_manager.cc:1397] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260926 07:33:20.680197 25042 raft_consensus.cc:348] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.680270 24522 catalog_manager.cc:4841] TS 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795): tablet 48a0f59f71e54395adab1be417638009 (table test-table [id=9af891a8da874cb49682b6d2f0ceb6ec]) successfully deleted
I20260926 07:33:20.680421 25042 raft_consensus.cc:374] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.680508 25042 raft_consensus.cc:729] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68c3391ff6743a88abfef8aa4ec866a, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.680763 25042 consensus_queue.cc:260] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.681435 25042 ts_tablet_manager.cc:1428] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:33:20.682595 25041 tablet_bootstrap.cc:492] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: No bootstrap required, opened a new log
I20260926 07:33:20.682766 25041 ts_tablet_manager.cc:1397] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20260926 07:33:20.683627 25041 raft_consensus.cc:348] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.683856 25041 raft_consensus.cc:374] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.683959 25041 raft_consensus.cc:729] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9627179fdc492fbb826ddd2e9824bb, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.684285 25041 consensus_queue.cc:260] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.687794 25041 ts_tablet_manager.cc:1428] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:33:20.693758 25066 ts_tablet_manager.cc:1918] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:20.693938 25066 log.cc:1196] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/48a0f59f71e54395adab1be417638009
I20260926 07:33:20.694489 25066 ts_tablet_manager.cc:1939] T 48a0f59f71e54395adab1be417638009 P 1c9627179fdc492fbb826ddd2e9824bb: Deleting consensus metadata
I20260926 07:33:20.696352 25067 tablet_replica.cc:330] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: stopping tablet replica
I20260926 07:33:20.696832 25067 raft_consensus.cc:2229] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:20.697129 25043 tablet_bootstrap.cc:492] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: No bootstrap required, opened a new log
I20260926 07:33:20.697319 25043 ts_tablet_manager.cc:1397] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.023s
I20260926 07:33:20.697335 25067 raft_consensus.cc:2258] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:20.698240 25043 raft_consensus.cc:348] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.698494 25043 raft_consensus.cc:374] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.698606 25043 raft_consensus.cc:729] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9bc7c57bf40efa28de91686331db8, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.698894 25043 consensus_queue.cc:260] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.699015 24522 catalog_manager.cc:4841] TS 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235): tablet 48a0f59f71e54395adab1be417638009 (table test-table [id=9af891a8da874cb49682b6d2f0ceb6ec]) successfully deleted
I20260926 07:33:20.699518 25043 ts_tablet_manager.cc:1428] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:33:20.700318 25067 ts_tablet_manager.cc:1905] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:20.705677 25067 ts_tablet_manager.cc:1918] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:20.705798 25067 log.cc:1196] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/48a0f59f71e54395adab1be417638009
I20260926 07:33:20.706254 25067 ts_tablet_manager.cc:1939] T 48a0f59f71e54395adab1be417638009 P c68c3391ff6743a88abfef8aa4ec866a: Deleting consensus metadata
I20260926 07:33:20.707728 24523 catalog_manager.cc:4841] TS c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907): tablet 48a0f59f71e54395adab1be417638009 (table test-table [id=9af891a8da874cb49682b6d2f0ceb6ec]) successfully deleted
I20260926 07:33:20.745782 25049 raft_consensus.cc:482] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:20.746008 25049 raft_consensus.cc:504] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.747670 25049 leader_election.cc:288] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795), 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235)
I20260926 07:33:20.754869 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.747943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:36795 (local address 127.22.221.67:41165)
0926 07:33:20.748163 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.748185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.748313 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.748862 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.748868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.748892 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.749131 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.749140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.750520 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.750529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.752611 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.752625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.754115 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.754122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.754316 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.754339 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.754446 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.754579 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":25}
I20260926 07:33:20.754906 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.748058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:41165 (local address 127.22.221.65:36795)
0926 07:33:20.748235 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.748241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.748260 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:33:20.748467 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.748476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.748588 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.748683 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.749306 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.750372 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.752803 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.753966 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.754145 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.754228 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.754472 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.754597 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:33:20.756053 25071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.748360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43235 (local address 127.22.221.67:59665)
0926 07:33:20.748964 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.748981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.749139 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.749539 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.749544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.749571 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:20.749891 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.749906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.751325 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.751331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.753421 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.753435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.755219 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.755249 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.755446 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.755483 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.755581 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.755712 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":4,"thread_start_us":479,"threads_started":1}
I20260926 07:33:20.756095 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.748671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:59665 (local address 127.22.221.66:43235)
0926 07:33:20.748850 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.748856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.749054 (+ 198us) server_negotiation.cc:407] Connection header received
0926 07:33:20.749294 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.749300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.749391 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.749497 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:20.750048 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.751187 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.753590 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.754837 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.754996 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.755066 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.755634 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.755786 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:33:20.756435 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d67bb7467734bfeb33a3a29f4d99063" candidate_uuid: "c68c3391ff6743a88abfef8aa4ec866a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9bc7c57bf40efa28de91686331db8" is_pre_election: true
I20260926 07:33:20.756942 24694 raft_consensus.cc:2454] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68c3391ff6743a88abfef8aa4ec866a in term 0.
I20260926 07:33:20.757644 24904 leader_election.cc:302] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9bc7c57bf40efa28de91686331db8, c68c3391ff6743a88abfef8aa4ec866a; no voters:
I20260926 07:33:20.757782 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d67bb7467734bfeb33a3a29f4d99063" candidate_uuid: "c68c3391ff6743a88abfef8aa4ec866a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9627179fdc492fbb826ddd2e9824bb" is_pre_election: true
I20260926 07:33:20.758013 24833 raft_consensus.cc:2454] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68c3391ff6743a88abfef8aa4ec866a in term 0.
I20260926 07:33:20.758109 25049 raft_consensus.cc:2781] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:20.758224 25049 raft_consensus.cc:482] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:20.758332 25049 raft_consensus.cc:3037] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:20.761574 25049 raft_consensus.cc:504] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.762297 25049 leader_election.cc:288] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [CANDIDATE]: Term 1 election: Requested vote from peers 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795), 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235)
I20260926 07:33:20.762802 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d67bb7467734bfeb33a3a29f4d99063" candidate_uuid: "c68c3391ff6743a88abfef8aa4ec866a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9bc7c57bf40efa28de91686331db8"
I20260926 07:33:20.762843 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d67bb7467734bfeb33a3a29f4d99063" candidate_uuid: "c68c3391ff6743a88abfef8aa4ec866a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9627179fdc492fbb826ddd2e9824bb"
I20260926 07:33:20.763028 24694 raft_consensus.cc:3037] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:20.763053 24833 raft_consensus.cc:3037] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:20.765074 24833 raft_consensus.cc:2454] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68c3391ff6743a88abfef8aa4ec866a in term 1.
I20260926 07:33:20.765497 24904 leader_election.cc:302] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9627179fdc492fbb826ddd2e9824bb, c68c3391ff6743a88abfef8aa4ec866a; no voters:
I20260926 07:33:20.765794 25049 raft_consensus.cc:2781] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:20.766263 25049 raft_consensus.cc:686] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 1 LEADER]: Becoming Leader. State: Replica: c68c3391ff6743a88abfef8aa4ec866a, State: Running, Role: LEADER
I20260926 07:33:20.766345 24694 raft_consensus.cc:2454] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68c3391ff6743a88abfef8aa4ec866a in term 1.
I20260926 07:33:20.766674 25049 consensus_queue.cc:237] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.770268 24536 catalog_manager.cc:5495] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a reported cstate change: term changed from 0 to 1, leader changed from <none> to c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67). New cstate: current_term: 1 leader_uuid: "c68c3391ff6743a88abfef8aa4ec866a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } health_report { overall_health: HEALTHY } } }
Sep 26 07:33:20 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407998, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:20.843932 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.829919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.22.221.126:37189)
0926 07:33:20.830141 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:20.830149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:20.830754 (+ 605us) server_negotiation.cc:407] Connection header received
0926 07:33:20.830985 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:20.830989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:20.831144 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:20.831264 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:20.832604 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.833348 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.834616 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.835057 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.835196 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.838651 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:20.838656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:20.838659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:20.838686 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:20.841198 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.841895 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.841900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.841905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.842030 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:20.842390 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:20.842396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:20.842399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:20.842674 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:20.842828 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:20.843266 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 07:33:20.843619 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":71}
I20260926 07:33:20.843971 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:20.829924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37189 (local address 127.0.0.1:57322)
0926 07:33:20.830602 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:20.830624 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:20.830792 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:20.831337 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:20.831341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:20.832174 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:20.832453 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.832462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.833484 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.833492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.834473 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:20.834484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:20.835261 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:20.835267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:20.835437 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:20.836140 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:20.836157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:20.838464 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:20.841328 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:20.841341 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:20.841349 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:20.841768 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:20.842137 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:20.842143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:20.842146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:20.842274 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:20.842802 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:20.842810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:20.842981 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 07:33:20.843152 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 07:33:20.843485 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":385,"threads_started":1}
I20260926 07:33:20.849279 24536 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:57322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:33:20.850591 24536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:20.857697 24674 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9745244a3849fd9ba7ec348ba56873 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=341b3ff7fc43449791118ba3bab9e9e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:33:20.858042 24813 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9745244a3849fd9ba7ec348ba56873 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=341b3ff7fc43449791118ba3bab9e9e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:33:20.858261 24674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9745244a3849fd9ba7ec348ba56873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.858567 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9745244a3849fd9ba7ec348ba56873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:20.863015 24952 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9745244a3849fd9ba7ec348ba56873 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=341b3ff7fc43449791118ba3bab9e9e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:33:20.863530 24952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9745244a3849fd9ba7ec348ba56873. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 07:33:20.867172 24745 tablet.cc:2369] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:33:20.869252 24884 tablet.cc:2369] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:20.871876 25041 tablet_bootstrap.cc:492] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: Bootstrap starting.
I20260926 07:33:20.872217 25042 tablet_bootstrap.cc:492] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: Bootstrap starting.
I20260926 07:33:20.872404 25043 tablet_bootstrap.cc:492] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: Bootstrap starting.
I20260926 07:33:20.873565 25041 tablet_bootstrap.cc:654] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.873934 25042 tablet_bootstrap.cc:654] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.874176 25043 tablet_bootstrap.cc:654] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:20.883605 25042 tablet_bootstrap.cc:492] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: No bootstrap required, opened a new log
I20260926 07:33:20.883809 25042 ts_tablet_manager.cc:1397] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260926 07:33:20.884626 25042 raft_consensus.cc:348] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.884913 25042 raft_consensus.cc:374] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 07:33:20.885159 25023 tablet.cc:2369] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:20.885424 25042 raft_consensus.cc:729] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68c3391ff6743a88abfef8aa4ec866a, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.885728 25042 consensus_queue.cc:260] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.886334 25042 ts_tablet_manager.cc:1428] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:20.886260 25049 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:20.886647 25049 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:20.888170 25041 tablet_bootstrap.cc:492] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: No bootstrap required, opened a new log
I20260926 07:33:20.888339 25041 ts_tablet_manager.cc:1397] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260926 07:33:20.889240 25041 raft_consensus.cc:348] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.889482 25041 raft_consensus.cc:374] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.889576 25041 raft_consensus.cc:729] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9627179fdc492fbb826ddd2e9824bb, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.889911 25041 consensus_queue.cc:260] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.890362 25048 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:20.890713 25048 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:20.891789 25041 ts_tablet_manager.cc:1428] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:33:20.900084 25043 tablet_bootstrap.cc:492] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: No bootstrap required, opened a new log
I20260926 07:33:20.900292 25043 ts_tablet_manager.cc:1397] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.015s
I20260926 07:33:20.901149 25043 raft_consensus.cc:348] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.901430 25043 raft_consensus.cc:374] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:20.901522 25043 raft_consensus.cc:729] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9bc7c57bf40efa28de91686331db8, State: Initialized, Role: FOLLOWER
I20260926 07:33:20.901795 25043 consensus_queue.cc:260] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:20.902472 25043 ts_tablet_manager.cc:1428] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:33:20.902312 25052 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:20.902693 25052 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:21.012606 25048 raft_consensus.cc:482] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:21.012832 25048 raft_consensus.cc:504] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:21.014458 25048 leader_election.cc:288] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795), c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907)
I20260926 07:33:21.022665 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.014911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:33855 (local address 127.22.221.67:40907)
0926 07:33:21.015109 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.015116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.016281 (+ 1165us) server_negotiation.cc:407] Connection header received
0926 07:33:21.016563 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.016569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.016676 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.016752 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:21.017595 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.018515 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.020329 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.021514 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.021676 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.021764 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.022190 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.022348 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:33:21.022902 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.014936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:58077 (local address 127.22.221.65:36795)
0926 07:33:21.015118 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.015125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.015813 (+ 688us) server_negotiation.cc:407] Connection header received
0926 07:33:21.016171 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.016176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.016269 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.016403 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:21.016986 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.018006 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.020594 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.021908 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.022098 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.022175 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.022472 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.022607 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:33:21.022938 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.014768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:36795 (local address 127.22.221.66:58077)
0926 07:33:21.015664 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:21.015690 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:21.016005 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:21.016469 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:21.016473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:21.016499 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:21.016828 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.016837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.018152 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.018157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.020422 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.020438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.022146 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.022152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.022310 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.022335 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:33:21.022441 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:33:21.022572 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":562,"threads_started":1}
I20260926 07:33:21.023469 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.015421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:40907 (local address 127.22.221.66:33855)
0926 07:33:21.016153 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:21.016172 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:21.016377 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:21.016973 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:21.016977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:21.017003 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:21.017399 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.017413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.018646 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.018656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.020172 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.020184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.021758 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.021765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.021962 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.022000 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:33:21.022951 (+ 951us) client_negotiation.cc:240] Negotiation successful
0926 07:33:21.023088 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":351,"threads_started":1}
I20260926 07:33:21.023865 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9745244a3849fd9ba7ec348ba56873" candidate_uuid: "1c9627179fdc492fbb826ddd2e9824bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9bc7c57bf40efa28de91686331db8" is_pre_election: true
I20260926 07:33:21.024159 24694 raft_consensus.cc:2454] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9627179fdc492fbb826ddd2e9824bb in term 0.
I20260926 07:33:21.024168 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9745244a3849fd9ba7ec348ba56873" candidate_uuid: "1c9627179fdc492fbb826ddd2e9824bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68c3391ff6743a88abfef8aa4ec866a" is_pre_election: true
I20260926 07:33:21.024432 24976 raft_consensus.cc:2454] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9627179fdc492fbb826ddd2e9824bb in term 0.
I20260926 07:33:21.024678 24765 leader_election.cc:302] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9627179fdc492fbb826ddd2e9824bb, 52f9bc7c57bf40efa28de91686331db8; no voters:
I20260926 07:33:21.025132 25048 raft_consensus.cc:2781] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:21.025270 25048 raft_consensus.cc:482] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:21.025378 25048 raft_consensus.cc:3037] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:21.027803 25048 raft_consensus.cc:504] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:21.028527 25048 leader_election.cc:288] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [CANDIDATE]: Term 1 election: Requested vote from peers 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795), c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907)
I20260926 07:33:21.028983 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9745244a3849fd9ba7ec348ba56873" candidate_uuid: "1c9627179fdc492fbb826ddd2e9824bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9bc7c57bf40efa28de91686331db8"
I20260926 07:33:21.029000 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9745244a3849fd9ba7ec348ba56873" candidate_uuid: "1c9627179fdc492fbb826ddd2e9824bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68c3391ff6743a88abfef8aa4ec866a"
I20260926 07:33:21.029201 24976 raft_consensus.cc:3037] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:21.029233 24694 raft_consensus.cc:3037] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:21.032210 24976 raft_consensus.cc:2454] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9627179fdc492fbb826ddd2e9824bb in term 1.
I20260926 07:33:21.032418 24694 raft_consensus.cc:2454] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9627179fdc492fbb826ddd2e9824bb in term 1.
I20260926 07:33:21.032671 24766 leader_election.cc:302] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9627179fdc492fbb826ddd2e9824bb, c68c3391ff6743a88abfef8aa4ec866a; no voters:
I20260926 07:33:21.033031 25048 raft_consensus.cc:2781] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:21.033607 25048 raft_consensus.cc:686] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 LEADER]: Becoming Leader. State: Replica: 1c9627179fdc492fbb826ddd2e9824bb, State: Running, Role: LEADER
I20260926 07:33:21.034013 25048 consensus_queue.cc:237] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } }
I20260926 07:33:21.036067 25085 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: TxnStatusTablet state changed. Reason: New leader 1c9627179fdc492fbb826ddd2e9824bb. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.036475 25085 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:33:21.037182 25087 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:33:21.037420 25087 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:33:21.038074 24536 catalog_manager.cc:5495] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66). New cstate: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } health_report { overall_health: UNKNOWN } } }
I20260926 07:33:21.081475 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.074081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.22.221.66:43235)
0926 07:33:21.074257 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.074263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.074329 (+ 66us) server_negotiation.cc:407] Connection header received
0926 07:33:21.074635 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.074641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.074787 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.074952 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:21.075364 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.076517 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.078474 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.079161 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.079328 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.079991 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.080788 (+ 797us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.080949 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.081064 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.081150 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:33:21.081516 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.073902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43235 (local address 127.0.0.1:45810)
0926 07:33:21.074248 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:21.074268 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:21.074416 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:21.075004 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:21.075008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:21.075021 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:21.075211 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.075221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.076702 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.076708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.078328 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.078342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.079359 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.079367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.079647 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.079744 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.080917 (+ 1173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.080939 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:33:21.081033 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:33:21.081211 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260926 07:33:21.086392 24976 raft_consensus.cc:1261] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Refusing update from remote peer 1c9627179fdc492fbb826ddd2e9824bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:21.086736 24694 raft_consensus.cc:1261] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Refusing update from remote peer 1c9627179fdc492fbb826ddd2e9824bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:21.087654 25048 consensus_queue.cc:1035] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:21.088093 25085 consensus_queue.cc:1035] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:21.095091 25049 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: TxnStatusTablet state changed. Reason: New leader 1c9627179fdc492fbb826ddd2e9824bb. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.095428 25049 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:21.099762 25052 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: TxnStatusTablet state changed. Reason: New leader 1c9627179fdc492fbb826ddd2e9824bb. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.100091 25052 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:21.101061 25049 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.101394 25049 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P c68c3391ff6743a88abfef8aa4ec866a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:21.105197 25048 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.105491 25048 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:33:21.106678 25085 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.106927 25085 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 1c9627179fdc492fbb826ddd2e9824bb: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:33:21.110623 25052 tablet_replica.cc:439] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c9627179fdc492fbb826ddd2e9824bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 } } peers { permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 } } peers { permanent_uuid: "c68c3391ff6743a88abfef8aa4ec866a" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40907 } } }
I20260926 07:33:21.110929 25052 tablet_replica.cc:442] T 6e9745244a3849fd9ba7ec348ba56873 P 52f9bc7c57bf40efa28de91686331db8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:21.130082 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.115343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.22.221.67:40907)
0926 07:33:21.115534 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.115540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.123966 (+ 8426us) server_negotiation.cc:407] Connection header received
0926 07:33:21.124175 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.124179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.124282 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.124461 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:21.125024 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.125743 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.127796 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.128272 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.128378 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.128884 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.129455 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.129571 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.129703 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.129776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:33:21.147980 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.140820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:58965 (local address 127.22.221.66:43235)
0926 07:33:21.141010 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.141014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.142629 (+ 1615us) server_negotiation.cc:407] Connection header received
0926 07:33:21.142858 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.142862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.142939 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.143060 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:21.143664 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.144469 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.145915 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.146547 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.146677 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.147117 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.147285 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.147390 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.147529 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.147610 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:33:21.148010 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.140836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43235 (local address 127.22.221.67:58965)
0926 07:33:21.142503 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:21.142519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:21.142709 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:21.143079 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:21.143083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:21.143095 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:21.143490 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.143501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.144602 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.144609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.145766 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.145778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.146734 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.146742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.146873 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.146928 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.147390 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.147411 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:33:21.147508 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:33:21.147639 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":1380,"threads_started":1}
I20260926 07:33:21.158794 24833 raft_consensus.cc:1261] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Refusing update from remote peer c68c3391ff6743a88abfef8aa4ec866a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:21.159072 24694 raft_consensus.cc:1261] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Refusing update from remote peer c68c3391ff6743a88abfef8aa4ec866a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:21.159574 25072 consensus_queue.cc:1035] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c9627179fdc492fbb826ddd2e9824bb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:21.160008 25049 consensus_queue.cc:1035] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f9bc7c57bf40efa28de91686331db8" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:21.210431 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.203490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:42497 (local address 127.22.221.67:40907)
0926 07:33:21.203776 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.203784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.204181 (+ 397us) server_negotiation.cc:407] Connection header received
0926 07:33:21.204406 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.204416 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.204485 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.204562 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:21.205241 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.206083 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.208500 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.208929 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.209029 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.209655 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.209771 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.209885 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.210026 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.210107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260926 07:33:21.210812 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.202963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:40907 (local address 127.22.221.66:42497)
0926 07:33:21.204059 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:21.204081 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:21.204232 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:21.204658 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:21.204663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:21.204677 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:21.205054 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.205067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.206467 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.206472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.208348 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:21.208364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.209109 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.209116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:21.209314 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.209399 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:21.209885 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:21.209897 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:33:21.209986 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:33:21.210145 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":739,"threads_started":1}
I20260926 07:33:21.250471 24537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:21.250746 24537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:21.254350 24537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693036aa441c4925b602bc23881a8fbe: Sending DeleteTablet for 3 replicas of tablet 4d67bb7467734bfeb33a3a29f4d99063
I20260926 07:33:21.255335 24674 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d67bb7467734bfeb33a3a29f4d99063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:21 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:56668
I20260926 07:33:21.255776 24952 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d67bb7467734bfeb33a3a29f4d99063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:21 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:35972
I20260926 07:33:21.256186 24813 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d67bb7467734bfeb33a3a29f4d99063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:21 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:45806
I20260926 07:33:21.263758 25107 tablet_replica.cc:330] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: stopping tablet replica
I20260926 07:33:21.264065 25107 raft_consensus.cc:2229] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:21.264417 25107 raft_consensus.cc:2258] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:21.266286 25107 ts_tablet_manager.cc:1905] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:21.270884 25106 tablet_replica.cc:330] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: stopping tablet replica
I20260926 07:33:21.271557 25106 raft_consensus.cc:2229] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:21.272194 25105 tablet_replica.cc:330] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: stopping tablet replica
I20260926 07:33:21.272420 25105 raft_consensus.cc:2229] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:21.272599 25106 raft_consensus.cc:2258] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:21.272656 25105 raft_consensus.cc:2258] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:21.273787 25106 ts_tablet_manager.cc:1905] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:21.274722 25105 ts_tablet_manager.cc:1905] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:21.281337 25106 ts_tablet_manager.cc:1918] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:21.281528 25106 log.cc:1196] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/4d67bb7467734bfeb33a3a29f4d99063
I20260926 07:33:21.282102 25106 ts_tablet_manager.cc:1939] T 4d67bb7467734bfeb33a3a29f4d99063 P c68c3391ff6743a88abfef8aa4ec866a: Deleting consensus metadata
I20260926 07:33:21.283342 24523 catalog_manager.cc:4841] TS c68c3391ff6743a88abfef8aa4ec866a (127.22.221.67:40907): tablet 4d67bb7467734bfeb33a3a29f4d99063 (table test-table [id=bbf188160df14670a69fa3e536299f26]) successfully deleted
I20260926 07:33:21.284446 25105 ts_tablet_manager.cc:1918] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:21.284593 25105 log.cc:1196] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/4d67bb7467734bfeb33a3a29f4d99063
I20260926 07:33:21.285049 25105 ts_tablet_manager.cc:1939] T 4d67bb7467734bfeb33a3a29f4d99063 P 52f9bc7c57bf40efa28de91686331db8: Deleting consensus metadata
I20260926 07:33:21.286181 24522 catalog_manager.cc:4841] TS 52f9bc7c57bf40efa28de91686331db8 (127.22.221.65:36795): tablet 4d67bb7467734bfeb33a3a29f4d99063 (table test-table [id=bbf188160df14670a69fa3e536299f26]) successfully deleted
I20260926 07:33:21.287757 25107 ts_tablet_manager.cc:1918] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:21.287933 25107 log.cc:1196] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/4d67bb7467734bfeb33a3a29f4d99063
I20260926 07:33:21.288455 25107 ts_tablet_manager.cc:1939] T 4d67bb7467734bfeb33a3a29f4d99063 P 1c9627179fdc492fbb826ddd2e9824bb: Deleting consensus metadata
I20260926 07:33:21.289667 24522 catalog_manager.cc:4841] TS 1c9627179fdc492fbb826ddd2e9824bb (127.22.221.66:43235): tablet 4d67bb7467734bfeb33a3a29f4d99063 (table test-table [id=bbf188160df14670a69fa3e536299f26]) successfully deleted
Sep 26 07:33:21 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:21.303438 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.268205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.22.221.65:36795)
0926 07:33:21.268392 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.268411 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.283293 (+ 14882us) server_negotiation.cc:407] Connection header received
0926 07:33:21.283627 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.283633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.283710 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.283776 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:21.286002 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.287111 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.289095 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.289782 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.289923 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.295325 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:21.295341 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:21.295362 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:21.295428 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:21.299458 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.300312 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.300320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.300324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.300472 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.300892 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.300897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.300900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.301316 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:21.301664 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.302796 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.303032 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 07:33:21.304505 24739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56686
I20260926 07:33:21.324700 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.307877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.22.221.126:37189)
0926 07:33:21.308109 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.308115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.308598 (+ 483us) server_negotiation.cc:407] Connection header received
0926 07:33:21.308851 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.308857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.308935 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.309025 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:21.310911 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.312071 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.313961 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.314585 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.314678 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.317975 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:21.317982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:21.317987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:21.318025 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:21.321549 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.322332 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.322337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.322340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.322508 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.323038 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.323042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.323045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.323405 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:21.323645 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.324141 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.324352 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 07:33:21.326005 24537 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326
I20260926 07:33:21.328104 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24608
I20260926 07:33:21.345670 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.330004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.22.221.65:36795)
0926 07:33:21.330185 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.330190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.330252 (+ 62us) server_negotiation.cc:407] Connection header received
0926 07:33:21.330512 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.330517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.330598 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.330668 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:21.332280 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.333343 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.335131 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.335807 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.335915 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.339129 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:21.339135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:21.339139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:21.339175 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:21.342614 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.343514 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.343521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.343524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.343686 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.344113 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.344121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.344126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.344458 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:21.344721 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.345062 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.345288 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:33:21.346231 24739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56702
W20260926 07:33:21.346807 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24608: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:21.346999 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24608
I20260926 07:33:21.377065 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24748
Sep 26 07:33:21 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:21.399294 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.378109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.22.221.66:43235)
0926 07:33:21.378308 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.378316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.378401 (+ 85us) server_negotiation.cc:407] Connection header received
0926 07:33:21.378615 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.378626 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.378720 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.378843 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:21.380426 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.381641 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.383772 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.384476 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.384581 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.390920 (+ 6339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:21.390935 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:21.390952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:21.390996 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:21.395602 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.396625 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.396634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.396638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.396795 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.397256 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.397262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.397268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.397736 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:21.398044 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.398623 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.398882 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 07:33:21.401116 24878 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45818
W20260926 07:33:21.401925 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24748: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:21.402114 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24748
I20260926 07:33:21.429404 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24887
Sep 26 07:33:21 dist-test-slave-qlq8 krb5kdc[24492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408000, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:21.447177 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:21.430135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.22.221.67:40907)
0926 07:33:21.430333 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:21.430339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:21.430360 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:33:21.430542 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:21.430548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:21.430630 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:21.430741 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:21.432000 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.433179 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.434492 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:21.435190 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:21.435385 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:21.439854 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:21.439870 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:21.439890 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:21.439951 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:21.444208 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.445008 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.445022 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.445027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.445192 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:21.445562 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:21.445567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:21.445571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:21.446058 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:21.446394 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:21.446562 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:33:21.446783 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 07:33:21.447996 25017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35978
W20260926 07:33:21.448643 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:21.448788 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24887
I20260926 07:33:21.468698 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 24505
I20260926 07:33:21.469714 24577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:21.585770 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 24505
2026-09-26T07:33:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4590 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:21 dist-test-slave-qlq8 krb5kdc[25116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:21 dist-test-slave-qlq8 krb5kdc[25116](info): set up 2 sockets
Sep 26 07:33:21 dist-test-slave-qlq8 krb5kdc[25116](info): commencing operation
krb5kdc: starting...
W20260926 07:33:22.675719 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:22 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:22.871346 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45547
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:45547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:23.190182 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:23.190546 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:23.190722 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:23.198102 25129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:23.198273 25129 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:33:23.198374 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:23.198500 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:23.198587 25129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:23.198675 25129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:23.209421 25129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:45547
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:23.211452 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:23.213673 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:23.226629 25134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:23.226714 25137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:23.226713 25135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:23.227626 25129 server_base.cc:1034] running on GCE node
I20260926 07:33:23.228422 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:23.229959 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:23.231312 25129 hybrid_clock.cc:648] HybridClock initialized: now 1790408003231198 us; error 44 us; skew 500 ppm
I20260926 07:33:23.231679 25129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:23 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:23.240101 25129 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:23.241690 25129 webserver.cc:460] Webserver started at http://127.22.221.126:43307/ using document root <none> and password file <none>
I20260926 07:33:23.242408 25129 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:23.242542 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:23.242863 25129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:23.245884 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "4dff99337bf64cb6a4476f1f4016762c"
format_stamp: "Formatted at 2026-09-26 07:33:23 on dist-test-slave-qlq8"
server_key: "7347ca4476ee55665a6062169293ddad"
server_key_iv: "0bb9abd1085c2809042e98429c07414a"
server_key_version: "encryptionkey@0"
I20260926 07:33:23.246709 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "4dff99337bf64cb6a4476f1f4016762c"
format_stamp: "Formatted at 2026-09-26 07:33:23 on dist-test-slave-qlq8"
server_key: "7347ca4476ee55665a6062169293ddad"
server_key_iv: "0bb9abd1085c2809042e98429c07414a"
server_key_version: "encryptionkey@0"
I20260926 07:33:23.251890 25129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:33:23.255335 25144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:23.256690 25129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:23.256871 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "4dff99337bf64cb6a4476f1f4016762c"
format_stamp: "Formatted at 2026-09-26 07:33:23 on dist-test-slave-qlq8"
server_key: "7347ca4476ee55665a6062169293ddad"
server_key_iv: "0bb9abd1085c2809042e98429c07414a"
server_key_version: "encryptionkey@0"
I20260926 07:33:23.257093 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:23.284664 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:23.288504 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:23.288772 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:23.310977 25129 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:45547
I20260926 07:33:23.310983 25205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:45547 every 8 connection(s)
I20260926 07:33:23.312341 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:23.316143 25206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:23.320559 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 25129
I20260926 07:33:23.320768 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:23.321600 23413 external_mini_cluster.cc:1428] Setting key 596de06e5cc47f4c704a483cb8b9f787
I20260926 07:33:23.327946 25206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Bootstrap starting.
I20260926 07:33:23.331664 25206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:23.332908 25206 log.cc:824] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:23.336282 25206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: No bootstrap required, opened a new log
Sep 26 07:33:23 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:23.346489 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:23.323686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.22.221.126:45547)
0926 07:33:23.324453 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:23.324464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:23.324503 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:33:23.326369 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:23.326392 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:23.326783 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:23.327095 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:23.328908 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.330200 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.332024 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.332727 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.332854 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:23.337650 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:23.337663 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:23.337682 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:23.337724 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:23.341738 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:23.342529 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:23.342540 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:23.342553 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:23.342767 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:23.343121 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:23.343127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:23.343131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:23.343903 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:23.344228 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:23.344552 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 07:33:23.345027 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":366,"threads_started":1}
I20260926 07:33:23.347167 25206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff99337bf64cb6a4476f1f4016762c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45547 } }
I20260926 07:33:23.347671 25206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:23.347805 25206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dff99337bf64cb6a4476f1f4016762c, State: Initialized, Role: FOLLOWER
I20260926 07:33:23.348419 25206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff99337bf64cb6a4476f1f4016762c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45547 } }
I20260926 07:33:23.348685 25206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:23.348848 25206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:23.349041 25206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:23.351293 25206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff99337bf64cb6a4476f1f4016762c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45547 } }
I20260926 07:33:23.351819 25206 leader_election.cc:302] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dff99337bf64cb6a4476f1f4016762c; no voters:
I20260926 07:33:23.352384 25206 leader_election.cc:288] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:23.352522 25211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:23.353242 25211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [term 1 LEADER]: Becoming Leader. State: Replica: 4dff99337bf64cb6a4476f1f4016762c, State: Running, Role: LEADER
I20260926 07:33:23.353811 25211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff99337bf64cb6a4476f1f4016762c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45547 } }
I20260926 07:33:23.354521 25206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:23.356643 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dff99337bf64cb6a4476f1f4016762c. Latest consensus state: current_term: 1 leader_uuid: "4dff99337bf64cb6a4476f1f4016762c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff99337bf64cb6a4476f1f4016762c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45547 } } }
I20260926 07:33:23.357095 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:23.357297 25212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dff99337bf64cb6a4476f1f4016762c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff99337bf64cb6a4476f1f4016762c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45547 } } }
I20260926 07:33:23.357592 25212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:23.358762 25216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:23.363981 25216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:23.374624 25216 catalog_manager.cc:1347] Generated new cluster ID: 5d2f4e5f744f43cea8a2e843d3762c3c
I20260926 07:33:23.374745 25216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:23.389887 25216 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:23.391678 25216 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:23.399806 25216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Generated new TSK 0
I20260926 07:33:23.400625 25216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:23.458943 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45547
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:23.798771 25232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:23.799044 25232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:23.799149 25232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:23.806145 25232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:23.806247 25232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:23.806329 25232 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:33:23.806387 25232 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:33:23.806456 25232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:23.815814 25232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:45547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:23.816979 25232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:23.818317 25232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:23.831437 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:23.831521 25238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:23.831473 25240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:23.832446 25232 server_base.cc:1034] running on GCE node
I20260926 07:33:23.833020 25232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:23.833901 25232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:23.835129 25232 hybrid_clock.cc:648] HybridClock initialized: now 1790408003835090 us; error 54 us; skew 500 ppm
I20260926 07:33:23.835470 25232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:23 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:23.838456 25232 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:23.839973 25232 webserver.cc:460] Webserver started at http://127.22.221.65:34251/ using document root <none> and password file <none>
I20260926 07:33:23.840591 25232 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:23.840687 25232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:23.840934 25232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:23.843809 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "2bf9278c33c046859c1ea531f8101487"
format_stamp: "Formatted at 2026-09-26 07:33:23 on dist-test-slave-qlq8"
server_key: "c35c3b47cda9ff28d991b1e2fe8d82a5"
server_key_iv: "c31f7c52cbd6ae58a3ee11b7aa72559b"
server_key_version: "encryptionkey@0"
I20260926 07:33:23.844503 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "2bf9278c33c046859c1ea531f8101487"
format_stamp: "Formatted at 2026-09-26 07:33:23 on dist-test-slave-qlq8"
server_key: "c35c3b47cda9ff28d991b1e2fe8d82a5"
server_key_iv: "c31f7c52cbd6ae58a3ee11b7aa72559b"
server_key_version: "encryptionkey@0"
I20260926 07:33:23.849617 25232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:33:23.852941 25247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:23.854311 25232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:33:23.854477 25232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "2bf9278c33c046859c1ea531f8101487"
format_stamp: "Formatted at 2026-09-26 07:33:23 on dist-test-slave-qlq8"
server_key: "c35c3b47cda9ff28d991b1e2fe8d82a5"
server_key_iv: "c31f7c52cbd6ae58a3ee11b7aa72559b"
server_key_version: "encryptionkey@0"
I20260926 07:33:23.854655 25232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:23.866400 25232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:23.870244 25232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:23.870529 25232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:23.873477 25232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:23.873570 25232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:23.873724 25232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:23.873842 25232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:33:23 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:23.899685 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:23.874135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:47329 (local address 127.22.221.126:45547)
0926 07:33:23.874708 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:23.874715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:23.876098 (+ 1383us) server_negotiation.cc:407] Connection header received
0926 07:33:23.878962 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:23.878972 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:23.879078 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:23.879171 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:23.882475 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.883296 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.885731 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.886385 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.886549 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:23.891579 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:23.891585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:23.891592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:23.891641 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:23.895156 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:23.896477 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:23.896487 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:23.896489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:23.896614 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:23.897248 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:23.897251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:23.897253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:23.897533 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:23.897638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:23.898927 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0926 07:33:23.899165 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":330,"threads_started":1}
I20260926 07:33:23.901728 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:23.874816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.22.221.65:47329)
0926 07:33:23.875737 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:23.875798 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:23.878537 (+ 2739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:23.879847 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:23.879879 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:23.880933 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:23.882211 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:23.882250 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.883471 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.883477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:23.885547 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:23.885560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.886910 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.886916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:23.887210 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:23.888080 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:23.888121 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:23.891329 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:23.895422 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:23.895437 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:23.895467 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:23.896304 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:23.896748 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:23.896757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:23.896763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:23.897123 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:23.897677 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:23.897690 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:23.898309 (+ 619us) client_negotiation.cc:769] Sending connection context
0926 07:33:23.898703 (+ 394us) client_negotiation.cc:240] Negotiation successful
0926 07:33:23.899488 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":398,"threads_started":1}
I20260926 07:33:23.924093 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:23.908483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.22.221.65:46009)
0926 07:33:23.909133 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:23.909149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:23.909293 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:23.909709 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:23.909712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:23.910157 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:23.910439 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:23.910448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.912155 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.912161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:23.913798 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:23.913815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.914959 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.914963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:23.915099 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:23.915844 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:23.915861 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:23.917164 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:23.921105 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:23.921112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:23.921119 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:23.921750 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:23.922224 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:23.922229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:23.922233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:23.922366 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:23.922899 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:23.922904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:23.923132 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 07:33:23.923337 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 07:33:23.923697 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":352,"threads_started":1}
I20260926 07:33:23.924218 25232 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:45901
I20260926 07:33:23.924268 25367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:45901 every 8 connection(s)
I20260926 07:33:23.924885 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:23.908385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:46009 (local address 127.22.221.126:45547)
0926 07:33:23.908633 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:23.908642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:23.909226 (+ 584us) server_negotiation.cc:407] Connection header received
0926 07:33:23.909453 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:23.909462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:23.909565 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:23.909686 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:23.910580 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.911967 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.914047 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:23.914591 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:23.914731 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:23.917384 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:23.917389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:23.917392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:23.917425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:23.920904 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:23.921967 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:23.921970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:23.921972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:23.922101 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:23.922491 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:23.922499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:23.922502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:23.922775 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:23.922951 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:23.924468 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0926 07:33:23.924663 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:33:23.925628 25232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:23.927878 25368 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45547
I20260926 07:33:23.928114 25368 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:23.928921 25368 heartbeater.cc:511] Master 127.22.221.126:45547 requested a full tablet report, sending...
I20260926 07:33:23.929968 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 25232
I20260926 07:33:23.930176 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:23.930862 23413 external_mini_cluster.cc:1428] Setting key e976116de783d502f3bb9bc8d4a7a88f
I20260926 07:33:23.931036 25161 ts_manager.cc:194] Registered new tserver with Master: 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901)
I20260926 07:33:23.932600 25161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:47329
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:23.973868 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45547
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:24.301995 25372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:24.302259 25372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:24.302369 25372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:24.309587 25372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:24.309698 25372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:24.309787 25372 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:33:24.309895 25372 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:33:24.309993 25372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:24.325201 25372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:45547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:24.326719 25372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:24.328456 25372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:24.342854 25377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.342793 25380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.342775 25378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.343778 25372 server_base.cc:1034] running on GCE node
I20260926 07:33:24.344404 25372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:24.345332 25372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:24.346568 25372 hybrid_clock.cc:648] HybridClock initialized: now 1790408004346520 us; error 42 us; skew 500 ppm
I20260926 07:33:24.346902 25372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:24 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:24.350116 25372 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:24.351611 25372 webserver.cc:460] Webserver started at http://127.22.221.66:39081/ using document root <none> and password file <none>
I20260926 07:33:24.352298 25372 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:24.352451 25372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:24.352771 25372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:24.355947 25372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "6ef65d6eb86d486a88fa245e76b093a6"
format_stamp: "Formatted at 2026-09-26 07:33:24 on dist-test-slave-qlq8"
server_key: "ef986c091191a203397f83f3e77e9986"
server_key_iv: "0b616bdc6225bb365503afa06fe654e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.356695 25372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "6ef65d6eb86d486a88fa245e76b093a6"
format_stamp: "Formatted at 2026-09-26 07:33:24 on dist-test-slave-qlq8"
server_key: "ef986c091191a203397f83f3e77e9986"
server_key_iv: "0b616bdc6225bb365503afa06fe654e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.362109 25372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:33:24.365491 25387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.366765 25372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:33:24.366941 25372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "6ef65d6eb86d486a88fa245e76b093a6"
format_stamp: "Formatted at 2026-09-26 07:33:24 on dist-test-slave-qlq8"
server_key: "ef986c091191a203397f83f3e77e9986"
server_key_iv: "0b616bdc6225bb365503afa06fe654e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.367141 25372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:24.379541 25372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:24.383343 25372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:24.383586 25372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:24.386116 25372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:24.386255 25372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.386394 25372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:24.386490 25372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:33:24 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:24.410463 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.386943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:53783 (local address 127.22.221.126:45547)
0926 07:33:24.387118 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.387124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.388728 (+ 1604us) server_negotiation.cc:407] Connection header received
0926 07:33:24.391350 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.391356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.391469 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.391612 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.393885 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.394992 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.397563 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.398007 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.398148 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.403133 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.403138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.403142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.403176 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.406380 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.407374 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.407381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.407386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.407568 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.408159 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.408167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.408171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.408550 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.408736 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.409888 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.410121 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:33:24.412149 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.387603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.22.221.66:53783)
0926 07:33:24.388537 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.388590 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.390959 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.391925 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.391933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.392669 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.393674 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.393712 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.395148 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.395155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.397372 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.397403 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.398253 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.398262 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.398582 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.399616 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.399651 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.402880 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.406532 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.406547 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.406566 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.407179 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.407709 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.407713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.407717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.407956 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.408835 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.408844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.409352 (+ 508us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.409752 (+ 400us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.410327 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":428,"threads_started":1}
I20260926 07:33:24.429032 25372 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:41739
I20260926 07:33:24.429068 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:41739 every 8 connection(s)
I20260926 07:33:24.430510 25372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:24.432866 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 25372
I20260926 07:33:24.433063 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:24.433409 25507 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45547
I20260926 07:33:24.433655 25507 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:24.433701 23413 external_mini_cluster.cc:1428] Setting key c5b246233bbb88291355a9d9cd54b3ac
I20260926 07:33:24.434386 25507 heartbeater.cc:511] Master 127.22.221.126:45547 requested a full tablet report, sending...
I20260926 07:33:24.435319 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.419664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.22.221.66:33223)
0926 07:33:24.420518 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.420547 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.420735 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.421145 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.421153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.421909 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.422555 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.422572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.424021 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.424028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.425785 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.425798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.426732 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.426739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.426919 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.427646 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.427664 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.428665 (+ 1001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.432216 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.432221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.432225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.432788 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.433386 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.433391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.433399 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.433548 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.434309 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.434314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.434486 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.434639 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.434938 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":443,"threads_started":1}
I20260926 07:33:24.436021 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.419589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:33223 (local address 127.22.221.126:45547)
0926 07:33:24.419829 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.419839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.420653 (+ 814us) server_negotiation.cc:407] Connection header received
0926 07:33:24.420880 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.420888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.420974 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.421072 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.422734 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.423888 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.426045 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.426518 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.426666 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.428852 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.428858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.428865 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.428905 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.432078 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.432933 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.432937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.432940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.433165 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.433688 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.433693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.433698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.434153 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.435359 (+ 1206us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.435493 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.435726 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:33:24.436393 25161 ts_manager.cc:194] Registered new tserver with Master: 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739)
I20260926 07:33:24.437288 25161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:53783
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:24.477962 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45547
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:24.796586 25511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:24.796867 25511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:24.796969 25511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:24.804140 25511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:24.804243 25511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:24.804387 25511 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:33:24.804469 25511 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:33:24.804600 25511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:24.814159 25511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:45547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:24.815599 25511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:24.816964 25511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:24.829887 25517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.829972 25516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.829952 25519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.830751 25511 server_base.cc:1034] running on GCE node
I20260926 07:33:24.831385 25511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:24.832227 25511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:24.833427 25511 hybrid_clock.cc:648] HybridClock initialized: now 1790408004833393 us; error 29 us; skew 500 ppm
I20260926 07:33:24.833756 25511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:24 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:24.836898 25511 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:24.838250 25511 webserver.cc:460] Webserver started at http://127.22.221.67:38419/ using document root <none> and password file <none>
I20260926 07:33:24.838941 25511 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:24.839072 25511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:24.839458 25511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:24.842325 25511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "d022a75c4d94425ca1e1aaba4787593f"
format_stamp: "Formatted at 2026-09-26 07:33:24 on dist-test-slave-qlq8"
server_key: "041079a3bdf28b7aa8fdaabde1e750a6"
server_key_iv: "1b8dcbb8d48d26669947ba26251a29b5"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.843107 25511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "d022a75c4d94425ca1e1aaba4787593f"
format_stamp: "Formatted at 2026-09-26 07:33:24 on dist-test-slave-qlq8"
server_key: "041079a3bdf28b7aa8fdaabde1e750a6"
server_key_iv: "1b8dcbb8d48d26669947ba26251a29b5"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.848058 25511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:33:24.851418 25526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.852649 25511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:24.852814 25511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "d022a75c4d94425ca1e1aaba4787593f"
format_stamp: "Formatted at 2026-09-26 07:33:24 on dist-test-slave-qlq8"
server_key: "041079a3bdf28b7aa8fdaabde1e750a6"
server_key_iv: "1b8dcbb8d48d26669947ba26251a29b5"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.853006 25511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:24.873765 25511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:24.878266 25511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:24.878556 25511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:24.881264 25511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:24.881407 25511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.881556 25511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:24.881639 25511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:33:24 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:24.904117 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.881644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:34855 (local address 127.22.221.126:45547)
0926 07:33:24.881890 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.881899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.883470 (+ 1571us) server_negotiation.cc:407] Connection header received
0926 07:33:24.885609 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.885618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.885726 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.885843 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.888091 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.889415 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.891011 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.891818 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.891975 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.896789 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.896800 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.896808 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.896874 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.900685 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.901689 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.901695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.901699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.901841 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.902293 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.902297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.902306 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.902579 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.902738 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.903524 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.903762 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:33:24.905233 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.882359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.22.221.67:34855)
0926 07:33:24.883268 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.883323 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.885299 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.886105 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.886119 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.886949 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.887866 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.887888 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.889590 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.889594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.890864 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.890875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.892120 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.892127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.892367 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.893256 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.893279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.896466 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.900877 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.900883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.900896 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.901547 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.901952 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.901956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.901960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.902169 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.902707 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.902714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.903069 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.903388 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.903957 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":395,"threads_started":1}
I20260926 07:33:24.927876 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.910866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:48773 (local address 127.22.221.126:45547)
0926 07:33:24.911060 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.911069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.911962 (+ 893us) server_negotiation.cc:407] Connection header received
0926 07:33:24.912149 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.912153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.912215 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.912296 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.913876 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.914839 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.917206 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.917735 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.917876 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.920664 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.920669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.920672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.920704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.924347 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.925457 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.925463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.925467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.925626 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.926131 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.926135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.926138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.926503 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.926663 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.927284 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.927524 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:33:24.928366 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.910901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.22.221.67:48773)
0926 07:33:24.911792 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.911825 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.912026 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.912417 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.912447 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.913231 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.913692 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.913708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.914971 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.914976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.917053 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.917068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.918101 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.918106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.918399 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.919136 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.919168 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.920405 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.924529 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.924538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.924545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.925296 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.925754 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.925762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.925769 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.925991 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.926632 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.926639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.926912 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.927092 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.927675 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":494,"threads_started":1}
I20260926 07:33:24.935366 25368 heartbeater.cc:503] Master 127.22.221.126:45547 was elected leader, sending a full tablet report...
I20260926 07:33:24.941607 25511 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:46809
I20260926 07:33:24.941644 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:46809 every 8 connection(s)
I20260926 07:33:24.942827 25511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:24.945011 25646 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45547
I20260926 07:33:24.945243 25646 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:24.945577 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 25511
I20260926 07:33:24.945722 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:24.945961 25646 heartbeater.cc:511] Master 127.22.221.126:45547 requested a full tablet report, sending...
I20260926 07:33:24.946123 23413 external_mini_cluster.cc:1428] Setting key 2e3a538997d8a15082d78097cbcd7a8c
I20260926 07:33:24.947475 25160 ts_manager.cc:194] Registered new tserver with Master: d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:24.948432 25160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:34855
I20260926 07:33:24.949115 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:33:24 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:33:24 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:24.996097 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.982498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.22.221.126:45547)
0926 07:33:24.982746 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.982753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.982962 (+ 209us) server_negotiation.cc:407] Connection header received
0926 07:33:24.983173 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.983181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.983303 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.983425 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.984744 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.985488 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.986707 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.987138 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.987292 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.991088 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.991096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.991105 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.991137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.993497 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.994095 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.994102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.994105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.994238 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.994604 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.994611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.994614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.994887 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.995148 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.995497 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.995722 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:33:24.999960 25160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:25.006855 25160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:25.041291 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.018761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.22.221.65:45901)
0926 07:33:25.031814 (+ 13053us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.031823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.031851 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:33:25.031951 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.031960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.032172 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.032414 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.033296 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.035212 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.038287 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.039606 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.039740 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.039966 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.040680 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.040969 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12889,"thread_start_us":12771,"threads_started":1}
I20260926 07:33:25.041376 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.018721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45901 (local address 127.0.0.1:49520)
0926 07:33:25.031163 (+ 12442us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.031181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.031320 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.032663 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.032670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.032705 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.033080 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.033094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.035453 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.035462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.038102 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.038119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.040116 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.040123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.040336 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.040466 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.040604 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.040816 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12329,"mutex_wait_us":57,"thread_start_us":12192,"threads_started":1}
I20260926 07:33:25.042318 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.018642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.22.221.67:46809)
0926 07:33:25.030921 (+ 12279us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.030932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.030970 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:33:25.031078 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.031086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.031449 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.031654 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.032706 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.034968 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.038913 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.040392 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.040514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.040746 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.041352 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.041972 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12074,"thread_start_us":11882,"threads_started":1}
I20260926 07:33:25.043285 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.018341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:46809 (local address 127.0.0.1:49126)
0926 07:33:25.025773 (+ 7432us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.025832 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.026067 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.031831 (+ 5764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.031840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.031896 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.032407 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.032423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.035183 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.035191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.038737 (+ 3546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.038751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.040922 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.040928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.041112 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.041145 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.042764 (+ 1619us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.042906 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7065,"mutex_wait_us":529,"thread_start_us":6900,"threads_started":1}
I20260926 07:33:25.045032 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.032069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41739 (local address 127.0.0.1:49136)
0926 07:33:25.036461 (+ 4392us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.036478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.036640 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.037482 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.037486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.037513 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.038202 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.038218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.039832 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.039839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.042226 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.042240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.044265 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.044272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.044441 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.044484 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.044589 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.044730 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4287,"thread_start_us":4312,"threads_started":1}
I20260926 07:33:25.045066 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.031672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.22.221.66:41739)
0926 07:33:25.036363 (+ 4691us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.036372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.036584 (+ 212us) server_negotiation.cc:407] Connection header received
0926 07:33:25.036809 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.036815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.037062 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.037253 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.038413 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.039654 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.042431 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.043822 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.043981 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.044209 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.044653 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.044811 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4526,"thread_start_us":4415,"threads_started":1}
I20260926 07:33:25.045562 25298 tablet_service.cc:1467] Processing CreateTablet for tablet ae067c13efd3428682eda3547ab9e75d (DEFAULT_TABLE table=test-table [id=a2c8ce59786545b18b1ec35f6f9d96b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:25.046819 25298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae067c13efd3428682eda3547ab9e75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.046813 25576 tablet_service.cc:1467] Processing CreateTablet for tablet ae067c13efd3428682eda3547ab9e75d (DEFAULT_TABLE table=test-table [id=a2c8ce59786545b18b1ec35f6f9d96b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:25.048131 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae067c13efd3428682eda3547ab9e75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.049630 25437 tablet_service.cc:1467] Processing CreateTablet for tablet ae067c13efd3428682eda3547ab9e75d (DEFAULT_TABLE table=test-table [id=a2c8ce59786545b18b1ec35f6f9d96b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:25.050915 25437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae067c13efd3428682eda3547ab9e75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.060195 25665 tablet_bootstrap.cc:492] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Bootstrap starting.
I20260926 07:33:25.062070 25666 tablet_bootstrap.cc:492] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Bootstrap starting.
I20260926 07:33:25.064271 25665 tablet_bootstrap.cc:654] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.065500 25666 tablet_bootstrap.cc:654] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.065716 25665 log.cc:824] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:25.066151 25667 tablet_bootstrap.cc:492] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Bootstrap starting.
I20260926 07:33:25.066813 25666 log.cc:824] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:25.069443 25665 tablet_bootstrap.cc:492] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: No bootstrap required, opened a new log
I20260926 07:33:25.069525 25667 tablet_bootstrap.cc:654] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.069720 25665 ts_tablet_manager.cc:1397] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 07:33:25.069744 25666 tablet_bootstrap.cc:492] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: No bootstrap required, opened a new log
I20260926 07:33:25.069998 25666 ts_tablet_manager.cc:1397] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 07:33:25.070763 25667 log.cc:824] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:25.073621 25667 tablet_bootstrap.cc:492] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: No bootstrap required, opened a new log
I20260926 07:33:25.073880 25667 ts_tablet_manager.cc:1397] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 07:33:25.077731 25665 raft_consensus.cc:348] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.078080 25665 raft_consensus.cc:374] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.078171 25665 raft_consensus.cc:729] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d022a75c4d94425ca1e1aaba4787593f, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.078806 25665 consensus_queue.cc:260] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.080030 25646 heartbeater.cc:503] Master 127.22.221.126:45547 was elected leader, sending a full tablet report...
I20260926 07:33:25.080574 25665 ts_tablet_manager.cc:1428] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260926 07:33:25.080416 25666 raft_consensus.cc:348] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.080808 25666 raft_consensus.cc:374] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.080919 25666 raft_consensus.cc:729] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.081449 25666 consensus_queue.cc:260] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.083026 25666 ts_tablet_manager.cc:1428] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260926 07:33:25.085069 25667 raft_consensus.cc:348] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.085477 25667 raft_consensus.cc:374] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.085568 25667 raft_consensus.cc:729] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.086135 25667 consensus_queue.cc:260] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.087286 25507 heartbeater.cc:503] Master 127.22.221.126:45547 was elected leader, sending a full tablet report...
I20260926 07:33:25.087726 25667 ts_tablet_manager.cc:1428] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
I20260926 07:33:25.125418 25672 raft_consensus.cc:482] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:25.125775 25672 raft_consensus.cc:504] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.127646 25672 leader_election.cc:288] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809), 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739)
I20260926 07:33:25.135483 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.127914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:39775 (local address 127.22.221.67:46809)
0926 07:33:25.128148 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.128157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.129287 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 07:33:25.129456 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.129462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.129555 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.129645 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.130325 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.131077 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.133675 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.134483 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.134660 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.134750 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.135060 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.135189 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:33:25.135516 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.128285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:57617 (local address 127.22.221.66:41739)
0926 07:33:25.128549 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.128558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.129432 (+ 874us) server_negotiation.cc:407] Connection header received
0926 07:33:25.129608 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.129611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.129707 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.129803 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.130442 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.131167 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.133743 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.134492 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.134684 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.134757 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.135102 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.135252 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:33:25.135641 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.127830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:46809 (local address 127.22.221.65:39775)
0926 07:33:25.129168 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.129184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.129300 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.129684 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.129689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.129714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.130155 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.130165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.131209 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.131217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.133518 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.133532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.134705 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.134712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.134888 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.134931 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.135006 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.135159 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":8,"thread_start_us":985,"threads_started":1}
I20260926 07:33:25.135691 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.128091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41739 (local address 127.22.221.65:57617)
0926 07:33:25.129338 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.129357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.129484 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.129838 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.129841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.129870 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.130243 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.130258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.131333 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.131337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.133570 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.133582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.134751 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.134757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.134902 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.134949 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.135058 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.135189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":80,"thread_start_us":882,"threads_started":1}
I20260926 07:33:25.136641 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae067c13efd3428682eda3547ab9e75d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f" is_pre_election: true
I20260926 07:33:25.136816 25457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae067c13efd3428682eda3547ab9e75d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef65d6eb86d486a88fa245e76b093a6" is_pre_election: true
I20260926 07:33:25.137279 25457 raft_consensus.cc:2454] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 0.
I20260926 07:33:25.137276 25601 raft_consensus.cc:2454] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 0.
I20260926 07:33:25.138015 25249 leader_election.cc:302] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, 6ef65d6eb86d486a88fa245e76b093a6; no voters:
I20260926 07:33:25.138442 25672 raft_consensus.cc:2781] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:25.138589 25672 raft_consensus.cc:482] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:25.138693 25672 raft_consensus.cc:3037] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.140924 25672 raft_consensus.cc:504] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.141513 25672 leader_election.cc:288] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 election: Requested vote from peers d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809), 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739)
I20260926 07:33:25.141969 25457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae067c13efd3428682eda3547ab9e75d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef65d6eb86d486a88fa245e76b093a6"
I20260926 07:33:25.141963 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae067c13efd3428682eda3547ab9e75d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f"
I20260926 07:33:25.142177 25601 raft_consensus.cc:3037] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.142177 25457 raft_consensus.cc:3037] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.145442 25457 raft_consensus.cc:2454] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 1.
I20260926 07:33:25.145442 25601 raft_consensus.cc:2454] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 1.
I20260926 07:33:25.145845 25251 leader_election.cc:302] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:25.146142 25672 raft_consensus.cc:2781] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:25.146593 25672 raft_consensus.cc:686] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Running, Role: LEADER
I20260926 07:33:25.147109 25672 consensus_queue.cc:237] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } }
I20260926 07:33:25.152032 25161 catalog_manager.cc:5495] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bf9278c33c046859c1ea531f8101487 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "2bf9278c33c046859c1ea531f8101487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } health_report { overall_health: UNKNOWN } } }
W20260926 07:33:25.179049 25369 tablet.cc:2369] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:33:25.182531 25508 tablet.cc:2369] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:33:25.194742 25647 tablet.cc:2369] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:33:25 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:25.220106 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.205794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45547 (local address 127.0.0.1:48332)
0926 07:33:25.206473 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.206490 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.206621 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.207049 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.207053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.207874 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:25.208167 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.208176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.209196 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.209200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.210219 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.210231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.211054 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.211060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.211201 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.211944 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:25.211965 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:25.214172 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:25.217221 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:25.217230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:25.217237 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:25.217653 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:25.218002 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:25.218005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:25.218008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:25.218148 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:25.218802 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:25.218812 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:25.219026 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.219280 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.219601 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":358,"threads_started":1}
I20260926 07:33:25.220351 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.205724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.22.221.126:45547)
0926 07:33:25.205952 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.205963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.206599 (+ 636us) server_negotiation.cc:407] Connection header received
0926 07:33:25.206830 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.206834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.206913 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.207006 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:25.208322 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.209067 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.210376 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.210832 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.210932 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.214388 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:25.214394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:25.214397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:25.214426 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:25.217081 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:25.217754 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:25.217758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:25.217761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:25.217901 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:25.218290 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:25.218293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:25.218295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:25.218633 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:25.218779 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.219388 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.219797 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":76}
I20260926 07:33:25.225890 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:48332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:33:25.226969 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:25.233722 25298 tablet_service.cc:1467] Processing CreateTablet for tablet de518b71d7bb46d7b9f3df1e1719a0af (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8461df4862bb4b25a708614101b73c1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:33:25.233808 25437 tablet_service.cc:1467] Processing CreateTablet for tablet de518b71d7bb46d7b9f3df1e1719a0af (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8461df4862bb4b25a708614101b73c1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:33:25.233990 25576 tablet_service.cc:1467] Processing CreateTablet for tablet de518b71d7bb46d7b9f3df1e1719a0af (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8461df4862bb4b25a708614101b73c1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:33:25.234740 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de518b71d7bb46d7b9f3df1e1719a0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.234735 25437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de518b71d7bb46d7b9f3df1e1719a0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.234746 25298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de518b71d7bb46d7b9f3df1e1719a0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.240233 25665 tablet_bootstrap.cc:492] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: Bootstrap starting.
I20260926 07:33:25.241680 25665 tablet_bootstrap.cc:654] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.241858 25667 tablet_bootstrap.cc:492] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: Bootstrap starting.
I20260926 07:33:25.241858 25666 tablet_bootstrap.cc:492] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: Bootstrap starting.
I20260926 07:33:25.243525 25666 tablet_bootstrap.cc:654] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.243525 25667 tablet_bootstrap.cc:654] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.247707 25666 tablet_bootstrap.cc:492] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: No bootstrap required, opened a new log
I20260926 07:33:25.247856 25667 tablet_bootstrap.cc:492] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: No bootstrap required, opened a new log
I20260926 07:33:25.247879 25666 ts_tablet_manager.cc:1397] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:33:25.248008 25667 ts_tablet_manager.cc:1397] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:33:25.248711 25666 raft_consensus.cc:348] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.248925 25666 raft_consensus.cc:374] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.248777 25667 raft_consensus.cc:348] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.249030 25667 raft_consensus.cc:374] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.249022 25666 raft_consensus.cc:729] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.249125 25667 raft_consensus.cc:729] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.249250 25666 consensus_queue.cc:260] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.249357 25667 consensus_queue.cc:260] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.249708 25672 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.249959 25667 ts_tablet_manager.cc:1428] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:33:25.249804 25673 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.250092 25666 ts_tablet_manager.cc:1428] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:25.250151 25673 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.250289 25672 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.250821 25665 tablet_bootstrap.cc:492] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: No bootstrap required, opened a new log
I20260926 07:33:25.250974 25665 ts_tablet_manager.cc:1397] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260926 07:33:25.251773 25665 raft_consensus.cc:348] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.252050 25665 raft_consensus.cc:374] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.252153 25665 raft_consensus.cc:729] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d022a75c4d94425ca1e1aaba4787593f, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.252409 25665 consensus_queue.cc:260] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.253016 25665 ts_tablet_manager.cc:1428] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:33:25.252895 25671 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.253265 25671 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.514518 25673 raft_consensus.cc:482] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:25.514798 25673 raft_consensus.cc:504] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.516491 25673 leader_election.cc:288] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:25.523288 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.516901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:50335 (local address 127.22.221.67:46809)
0926 07:33:25.517159 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.517168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.517721 (+ 553us) server_negotiation.cc:407] Connection header received
0926 07:33:25.517922 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.517931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.518020 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.518098 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.518833 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.519643 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.521536 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.522279 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.522445 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.522533 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.522867 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.522998 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:33:25.523308 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.516945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:54991 (local address 127.22.221.65:45901)
0926 07:33:25.517230 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.517237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.517833 (+ 596us) server_negotiation.cc:407] Connection header received
0926 07:33:25.518017 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.518021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.518120 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.518212 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.518861 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.519608 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.521495 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.522340 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.522522 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.522596 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.522900 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.523022 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 07:33:25.523607 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.516790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45901 (local address 127.22.221.66:54991)
0926 07:33:25.517700 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.517720 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.517887 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.518252 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.518257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.518286 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.518674 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.518683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.519742 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.519750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.521324 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.521337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.522599 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.522604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.522722 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.522743 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.522855 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.522986 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":20,"thread_start_us":556,"threads_started":1}
I20260926 07:33:25.523733 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.516771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:46809 (local address 127.22.221.66:50335)
0926 07:33:25.517587 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.517613 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.517747 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.518148 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.518155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.518242 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.518641 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.518654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.519807 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.519811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.521343 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.521355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.522508 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.522514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.522701 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.522725 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.522802 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.522950 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":35,"thread_start_us":474,"threads_started":1}
I20260926 07:33:25.524355 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de518b71d7bb46d7b9f3df1e1719a0af" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f" is_pre_election: true
I20260926 07:33:25.524614 25601 raft_consensus.cc:2454] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 0.
I20260926 07:33:25.524605 25318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de518b71d7bb46d7b9f3df1e1719a0af" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf9278c33c046859c1ea531f8101487" is_pre_election: true
I20260926 07:33:25.525080 25318 raft_consensus.cc:2454] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 0.
I20260926 07:33:25.525161 25391 leader_election.cc:302] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef65d6eb86d486a88fa245e76b093a6, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:25.525787 25673 raft_consensus.cc:2781] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:25.525941 25673 raft_consensus.cc:482] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:25.526047 25673 raft_consensus.cc:3037] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.528215 25673 raft_consensus.cc:504] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.528789 25673 leader_election.cc:288] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 election: Requested vote from peers 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:25.529247 25318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de518b71d7bb46d7b9f3df1e1719a0af" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf9278c33c046859c1ea531f8101487"
I20260926 07:33:25.529336 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de518b71d7bb46d7b9f3df1e1719a0af" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f"
I20260926 07:33:25.529482 25318 raft_consensus.cc:3037] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.529541 25601 raft_consensus.cc:3037] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.532496 25601 raft_consensus.cc:2454] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 1.
I20260926 07:33:25.532509 25318 raft_consensus.cc:2454] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 1.
I20260926 07:33:25.532948 25391 leader_election.cc:302] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef65d6eb86d486a88fa245e76b093a6, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:25.533299 25673 raft_consensus.cc:2781] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:25.533828 25673 raft_consensus.cc:686] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Running, Role: LEADER
I20260926 07:33:25.534355 25673 consensus_queue.cc:237] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.537710 25689 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: TxnStatusTablet state changed. Reason: New leader 6ef65d6eb86d486a88fa245e76b093a6. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.538121 25689 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:33:25.538864 25691 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:33:25.539147 25691 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:33:25.539150 25161 catalog_manager.cc:5495] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66). New cstate: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260926 07:33:25.544620 25672 consensus_queue.cc:1035] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:25.551704 25676 consensus_queue.cc:1035] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:25.619021 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.613236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41739 (local address 127.0.0.1:49140)
0926 07:33:25.613501 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.613522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.613675 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.614174 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.614179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.614193 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:25.614389 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.614401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.615595 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.615600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.616598 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.616611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.617398 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.617404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.617586 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.617639 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.618477 (+ 838us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.618500 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.618592 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.618743 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:33:25.619061 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.613389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.22.221.66:41739)
0926 07:33:25.613535 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.613540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.613576 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:33:25.613927 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.613931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.614005 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.614123 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:25.614528 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.615438 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.616759 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.617221 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.617374 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.617846 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.618356 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.618531 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.618615 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.618695 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:33:25.625530 25601 raft_consensus.cc:1261] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Refusing update from remote peer 6ef65d6eb86d486a88fa245e76b093a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:25.625845 25318 raft_consensus.cc:1261] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Refusing update from remote peer 6ef65d6eb86d486a88fa245e76b093a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:25.626204 25689 consensus_queue.cc:1035] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:25.626693 25673 consensus_queue.cc:1035] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:25.632102 25672 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: TxnStatusTablet state changed. Reason: New leader 6ef65d6eb86d486a88fa245e76b093a6. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.632472 25672 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.634338 25671 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: TxnStatusTablet state changed. Reason: New leader 6ef65d6eb86d486a88fa245e76b093a6. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.634599 25671 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.635212 25695 mvcc.cc:205] Tried to move back new op lower bound from 7333511191034494976 to 7333511190672703488. Current Snapshot: MvccSnapshot[applied={T|T < 7333511191034494976}]
I20260926 07:33:25.644029 25689 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.644295 25689 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:33:25.646894 25689 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.647148 25689 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 6ef65d6eb86d486a88fa245e76b093a6: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:33:25.650106 25672 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.650166 25671 tablet_replica.cc:439] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } } }
I20260926 07:33:25.650357 25672 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P 2bf9278c33c046859c1ea531f8101487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.650398 25671 tablet_replica.cc:442] T de518b71d7bb46d7b9f3df1e1719a0af P d022a75c4d94425ca1e1aaba4787593f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:33:25.662757 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.655953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.22.221.65:45901)
0926 07:33:25.656132 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.656140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.656683 (+ 543us) server_negotiation.cc:407] Connection header received
0926 07:33:25.656961 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.656967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.657048 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.657167 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:25.657644 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.658541 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.660541 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.660969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.661067 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.661616 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.662148 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.662257 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.662389 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.662452 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:33:25.678721 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.672874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41739 (local address 127.22.221.65:49619)
0926 07:33:25.673495 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.673511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.673617 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.673959 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.673962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.673971 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:25.674183 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.674190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.675595 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.675599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.676659 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.676672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.677670 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.677675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.677798 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.677849 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.678269 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.678277 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.678343 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.678443 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":321,"threads_started":1}
I20260926 07:33:25.678762 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.672925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:49619 (local address 127.22.221.66:41739)
0926 07:33:25.673144 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.673151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.673590 (+ 439us) server_negotiation.cc:407] Connection header received
0926 07:33:25.673734 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.673742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.673845 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.673943 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:25.674319 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.675471 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.676813 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.677509 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.677639 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.678040 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.678173 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.678265 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.678388 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.678471 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:33:25.724200 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.713431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:34147 (local address 127.22.221.65:45901)
0926 07:33:25.713665 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.713673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.718497 (+ 4824us) server_negotiation.cc:407] Connection header received
0926 07:33:25.718712 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.718717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.718814 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.718911 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:25.719679 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.720431 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.722090 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.722689 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.722823 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.723443 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.723575 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.723666 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.723821 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.723898 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:33:25.724331 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.713497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45901 (local address 127.22.221.66:34147)
0926 07:33:25.718378 (+ 4881us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.718399 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.718555 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.718955 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.718959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.718971 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:25.719488 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.719505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.720568 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.720574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.721963 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.721980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.722890 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.722897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.723126 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.723200 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.723689 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.723701 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.723772 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.723925 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4643,"thread_start_us":4504,"threads_started":1}
I20260926 07:33:25.761482 25160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:25.761796 25160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:25.765882 25160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Sending DeleteTablet for 3 replicas of tablet ae067c13efd3428682eda3547ab9e75d
I20260926 07:33:25.767036 25576 tablet_service.cc:1514] Processing DeleteTablet for tablet ae067c13efd3428682eda3547ab9e75d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:25 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49126
I20260926 07:33:25.767632 25437 tablet_service.cc:1514] Processing DeleteTablet for tablet ae067c13efd3428682eda3547ab9e75d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:25 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49136
I20260926 07:33:25.767892 25298 tablet_service.cc:1514] Processing DeleteTablet for tablet ae067c13efd3428682eda3547ab9e75d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:25 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49520
I20260926 07:33:25.770015 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:25.770725 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:25.779052 25436 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec20454870a4f10a7d8a1862dd25aaf (DEFAULT_TABLE table=test-table [id=0d06d74748624233bb0f39cffaf6a67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:25.779680 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec20454870a4f10a7d8a1862dd25aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.779814 25716 tablet_replica.cc:330] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: stopping tablet replica
I20260926 07:33:25.780076 25717 tablet_replica.cc:330] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: stopping tablet replica
I20260926 07:33:25.780772 25297 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec20454870a4f10a7d8a1862dd25aaf (DEFAULT_TABLE table=test-table [id=0d06d74748624233bb0f39cffaf6a67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:25.780941 25717 raft_consensus.cc:2229] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:25.781153 25716 raft_consensus.cc:2229] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:25.781574 25716 raft_consensus.cc:2258] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:25.781642 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec20454870a4f10a7d8a1862dd25aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.783442 25575 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec20454870a4f10a7d8a1862dd25aaf (DEFAULT_TABLE table=test-table [id=0d06d74748624233bb0f39cffaf6a67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:25.783619 25716 ts_tablet_manager.cc:1905] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:25.783916 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec20454870a4f10a7d8a1862dd25aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:25.790858 25716 ts_tablet_manager.cc:1918] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:25.791042 25716 log.cc:1196] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/ae067c13efd3428682eda3547ab9e75d
I20260926 07:33:25.791611 25716 ts_tablet_manager.cc:1939] T ae067c13efd3428682eda3547ab9e75d P 6ef65d6eb86d486a88fa245e76b093a6: Deleting consensus metadata
I20260926 07:33:25.793488 25146 catalog_manager.cc:4841] TS 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739): tablet ae067c13efd3428682eda3547ab9e75d (table test-table [id=a2c8ce59786545b18b1ec35f6f9d96b5]) successfully deleted
I20260926 07:33:25.795725 25717 raft_consensus.cc:2258] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:25.797168 25715 tablet_replica.cc:330] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: stopping tablet replica
I20260926 07:33:25.797287 25717 ts_tablet_manager.cc:1905] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:25.797605 25715 raft_consensus.cc:2229] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:25.797948 25715 raft_consensus.cc:2258] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:25.799813 25715 ts_tablet_manager.cc:1905] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:25.805399 25718 tablet_bootstrap.cc:492] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Bootstrap starting.
I20260926 07:33:25.806296 25717 ts_tablet_manager.cc:1918] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:25.806517 25717 log.cc:1196] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/ae067c13efd3428682eda3547ab9e75d
I20260926 07:33:25.807179 25717 ts_tablet_manager.cc:1939] T ae067c13efd3428682eda3547ab9e75d P 2bf9278c33c046859c1ea531f8101487: Deleting consensus metadata
I20260926 07:33:25.808251 25718 tablet_bootstrap.cc:654] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.808689 25715 ts_tablet_manager.cc:1918] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:25.808880 25715 log.cc:1196] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/ae067c13efd3428682eda3547ab9e75d
I20260926 07:33:25.809242 25146 catalog_manager.cc:4841] TS 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901): tablet ae067c13efd3428682eda3547ab9e75d (table test-table [id=a2c8ce59786545b18b1ec35f6f9d96b5]) successfully deleted
I20260926 07:33:25.809505 25715 ts_tablet_manager.cc:1939] T ae067c13efd3428682eda3547ab9e75d P d022a75c4d94425ca1e1aaba4787593f: Deleting consensus metadata
I20260926 07:33:25.809824 25720 tablet_bootstrap.cc:492] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Bootstrap starting.
I20260926 07:33:25.811254 25148 catalog_manager.cc:4841] TS d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809): tablet ae067c13efd3428682eda3547ab9e75d (table test-table [id=a2c8ce59786545b18b1ec35f6f9d96b5]) successfully deleted
I20260926 07:33:25.811633 25720 tablet_bootstrap.cc:654] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.812788 25719 tablet_bootstrap.cc:492] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Bootstrap starting.
I20260926 07:33:25.814143 25719 tablet_bootstrap.cc:654] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:25.820851 25720 tablet_bootstrap.cc:492] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: No bootstrap required, opened a new log
I20260926 07:33:25.821559 25719 tablet_bootstrap.cc:492] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: No bootstrap required, opened a new log
I20260926 07:33:25.821738 25719 ts_tablet_manager.cc:1397] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260926 07:33:25.821923 25720 ts_tablet_manager.cc:1397] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260926 07:33:25.822745 25719 raft_consensus.cc:348] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.823702 25720 raft_consensus.cc:348] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.823980 25720 raft_consensus.cc:374] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.824115 25719 raft_consensus.cc:374] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.824244 25720 raft_consensus.cc:729] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d022a75c4d94425ca1e1aaba4787593f, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.824384 25719 raft_consensus.cc:729] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.824779 25720 consensus_queue.cc:260] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.826325 25719 consensus_queue.cc:260] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.829849 25720 ts_tablet_manager.cc:1428] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260926 07:33:25.832477 25719 ts_tablet_manager.cc:1428] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260926 07:33:25.840062 25718 tablet_bootstrap.cc:492] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: No bootstrap required, opened a new log
I20260926 07:33:25.840252 25718 ts_tablet_manager.cc:1397] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.022s
I20260926 07:33:25.841094 25718 raft_consensus.cc:348] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.841356 25718 raft_consensus.cc:374] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:25.841470 25718 raft_consensus.cc:729] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Initialized, Role: FOLLOWER
I20260926 07:33:25.841785 25718 consensus_queue.cc:260] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:25.842574 25718 ts_tablet_manager.cc:1428] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:33:26.099888 25676 raft_consensus.cc:482] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:26.100152 25676 raft_consensus.cc:504] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:26.100921 25676 leader_election.cc:288] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:26.101526 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec20454870a4f10a7d8a1862dd25aaf" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f" is_pre_election: true
I20260926 07:33:26.101538 25457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec20454870a4f10a7d8a1862dd25aaf" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef65d6eb86d486a88fa245e76b093a6" is_pre_election: true
I20260926 07:33:26.101776 25601 raft_consensus.cc:2454] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 0.
I20260926 07:33:26.101825 25457 raft_consensus.cc:2454] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 0.
I20260926 07:33:26.102262 25251 leader_election.cc:302] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:26.102595 25676 raft_consensus.cc:2781] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:26.102730 25676 raft_consensus.cc:482] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:26.102847 25676 raft_consensus.cc:3037] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:26.106146 25676 raft_consensus.cc:504] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:26.107136 25676 leader_election.cc:288] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 election: Requested vote from peers 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:26.107391 25457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec20454870a4f10a7d8a1862dd25aaf" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef65d6eb86d486a88fa245e76b093a6"
I20260926 07:33:26.107460 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec20454870a4f10a7d8a1862dd25aaf" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f"
I20260926 07:33:26.107671 25457 raft_consensus.cc:3037] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:26.107713 25601 raft_consensus.cc:3037] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:26.110036 25457 raft_consensus.cc:2454] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 1.
I20260926 07:33:26.110466 25249 leader_election.cc:302] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, 6ef65d6eb86d486a88fa245e76b093a6; no voters:
I20260926 07:33:26.110623 25601 raft_consensus.cc:2454] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 1.
I20260926 07:33:26.110741 25676 raft_consensus.cc:2781] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:26.110922 25676 raft_consensus.cc:686] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Running, Role: LEADER
I20260926 07:33:26.111505 25676 consensus_queue.cc:237] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:26.114648 25161 catalog_manager.cc:5495] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bf9278c33c046859c1ea531f8101487 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "2bf9278c33c046859c1ea531f8101487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260926 07:33:26.174304 25457 raft_consensus.cc:1261] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Refusing update from remote peer 2bf9278c33c046859c1ea531f8101487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:26.174917 25601 raft_consensus.cc:1261] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Refusing update from remote peer 2bf9278c33c046859c1ea531f8101487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:26.174947 25676 consensus_queue.cc:1035] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:26.175680 25672 consensus_queue.cc:1035] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:26.195127 25697 mvcc.cc:205] Tried to move back new op lower bound from 7333511193284612096 to 7333511193035431936. Current Snapshot: MvccSnapshot[applied={T|T < 7333511193284612096}]
W20260926 07:33:26.201148 25694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:33:27.787894 25161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:27.788179 25161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:27.792584 25161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Sending DeleteTablet for 3 replicas of tablet 5ec20454870a4f10a7d8a1862dd25aaf
I20260926 07:33:27.793393 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec20454870a4f10a7d8a1862dd25aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:27 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49520
I20260926 07:33:27.793486 25436 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec20454870a4f10a7d8a1862dd25aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:27 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49136
I20260926 07:33:27.793617 25575 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec20454870a4f10a7d8a1862dd25aaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:27 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49126
I20260926 07:33:27.806172 25749 tablet_replica.cc:330] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: stopping tablet replica
I20260926 07:33:27.806461 25750 tablet_replica.cc:330] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: stopping tablet replica
I20260926 07:33:27.806659 25751 tablet_replica.cc:330] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: stopping tablet replica
I20260926 07:33:27.806674 25749 raft_consensus.cc:2229] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:27.806746 25750 raft_consensus.cc:2229] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:27.806931 25751 raft_consensus.cc:2229] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:27.807051 25750 raft_consensus.cc:2258] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:27.807286 25751 raft_consensus.cc:2258] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:27.807204 25749 raft_consensus.cc:2258] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:27.808344 25751 ts_tablet_manager.cc:1905] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:27.808358 25750 ts_tablet_manager.cc:1905] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:27.808626 25749 ts_tablet_manager.cc:1905] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:27.815112 25750 ts_tablet_manager.cc:1918] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:33:27.815174 25751 ts_tablet_manager.cc:1918] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:33:27.815335 25751 log.cc:1196] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/5ec20454870a4f10a7d8a1862dd25aaf
I20260926 07:33:27.815369 25750 log.cc:1196] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/5ec20454870a4f10a7d8a1862dd25aaf
I20260926 07:33:27.815867 25751 ts_tablet_manager.cc:1939] T 5ec20454870a4f10a7d8a1862dd25aaf P d022a75c4d94425ca1e1aaba4787593f: Deleting consensus metadata
I20260926 07:33:27.815872 25750 ts_tablet_manager.cc:1939] T 5ec20454870a4f10a7d8a1862dd25aaf P 6ef65d6eb86d486a88fa245e76b093a6: Deleting consensus metadata
I20260926 07:33:27.816413 25749 ts_tablet_manager.cc:1918] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:33:27.816601 25749 log.cc:1196] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/5ec20454870a4f10a7d8a1862dd25aaf
I20260926 07:33:27.816980 25146 catalog_manager.cc:4841] TS 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739): tablet 5ec20454870a4f10a7d8a1862dd25aaf (table test-table [id=0d06d74748624233bb0f39cffaf6a67e]) successfully deleted
I20260926 07:33:27.817006 25148 catalog_manager.cc:4841] TS d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809): tablet 5ec20454870a4f10a7d8a1862dd25aaf (table test-table [id=0d06d74748624233bb0f39cffaf6a67e]) successfully deleted
I20260926 07:33:27.817463 25749 ts_tablet_manager.cc:1939] T 5ec20454870a4f10a7d8a1862dd25aaf P 2bf9278c33c046859c1ea531f8101487: Deleting consensus metadata
I20260926 07:33:27.818588 25146 catalog_manager.cc:4841] TS 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901): tablet 5ec20454870a4f10a7d8a1862dd25aaf (table test-table [id=0d06d74748624233bb0f39cffaf6a67e]) successfully deleted
Sep 26 07:33:27 dist-test-slave-qlq8 krb5kdc[25116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:33:27 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:27.850616 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:27.837143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.22.221.126:45547)
0926 07:33:27.837702 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:27.837707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:27.837729 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:33:27.837891 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:27.837895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:27.837960 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:27.838057 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:27.839352 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:27.840135 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:27.841352 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:27.841745 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:27.841849 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:27.845754 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:27.845759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:27.845762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:27.845796 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:27.848399 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:27.848987 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:27.848990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:27.848992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:27.849113 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:27.849433 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:27.849436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:27.849437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:27.849690 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:27.849874 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:27.850084 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 07:33:27.850240 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":300,"threads_started":1}
I20260926 07:33:27.854375 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:27.854961 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:27.861931 25297 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99331be41540c28a5cd2fb6992f825 (DEFAULT_TABLE table=test-table [id=4a6077c8b3e04877bdb41a7a5fc9682e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:27.862056 25575 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99331be41540c28a5cd2fb6992f825 (DEFAULT_TABLE table=test-table [id=4a6077c8b3e04877bdb41a7a5fc9682e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:27.862419 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99331be41540c28a5cd2fb6992f825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:27.862555 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99331be41540c28a5cd2fb6992f825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:27.867676 25436 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99331be41540c28a5cd2fb6992f825 (DEFAULT_TABLE table=test-table [id=4a6077c8b3e04877bdb41a7a5fc9682e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:27.868309 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99331be41540c28a5cd2fb6992f825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:27.877027 25761 tablet_bootstrap.cc:492] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Bootstrap starting.
I20260926 07:33:27.877669 25762 tablet_bootstrap.cc:492] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Bootstrap starting.
I20260926 07:33:27.878569 25760 tablet_bootstrap.cc:492] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Bootstrap starting.
I20260926 07:33:27.878921 25761 tablet_bootstrap.cc:654] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:27.879483 25762 tablet_bootstrap.cc:654] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:27.880323 25760 tablet_bootstrap.cc:654] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:27.881660 25761 tablet_bootstrap.cc:492] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: No bootstrap required, opened a new log
I20260926 07:33:27.881860 25761 ts_tablet_manager.cc:1397] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 07:33:27.882972 25761 raft_consensus.cc:348] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.883329 25761 raft_consensus.cc:374] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:27.883505 25761 raft_consensus.cc:729] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Initialized, Role: FOLLOWER
I20260926 07:33:27.883843 25761 consensus_queue.cc:260] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.884969 25761 ts_tablet_manager.cc:1428] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:27.886888 25762 tablet_bootstrap.cc:492] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: No bootstrap required, opened a new log
I20260926 07:33:27.887058 25762 ts_tablet_manager.cc:1397] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260926 07:33:27.887578 25760 tablet_bootstrap.cc:492] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: No bootstrap required, opened a new log
I20260926 07:33:27.887969 25760 ts_tablet_manager.cc:1397] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260926 07:33:27.887985 25762 raft_consensus.cc:348] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.888279 25762 raft_consensus.cc:374] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:27.888386 25762 raft_consensus.cc:729] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Initialized, Role: FOLLOWER
I20260926 07:33:27.888684 25762 consensus_queue.cc:260] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.888823 25760 raft_consensus.cc:348] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.889145 25760 raft_consensus.cc:374] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:27.889287 25760 raft_consensus.cc:729] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d022a75c4d94425ca1e1aaba4787593f, State: Initialized, Role: FOLLOWER
I20260926 07:33:27.889348 25762 ts_tablet_manager.cc:1428] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:27.889580 25760 consensus_queue.cc:260] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.890420 25760 ts_tablet_manager.cc:1428] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:27.962155 25673 raft_consensus.cc:482] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:27.962363 25673 raft_consensus.cc:504] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.963122 25673 leader_election.cc:288] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:27.963691 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99331be41540c28a5cd2fb6992f825" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f" is_pre_election: true
I20260926 07:33:27.963675 25318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99331be41540c28a5cd2fb6992f825" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf9278c33c046859c1ea531f8101487" is_pre_election: true
I20260926 07:33:27.963930 25318 raft_consensus.cc:2454] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 0.
I20260926 07:33:27.963953 25601 raft_consensus.cc:2454] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 0.
I20260926 07:33:27.964440 25389 leader_election.cc:302] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, 6ef65d6eb86d486a88fa245e76b093a6; no voters:
I20260926 07:33:27.964761 25673 raft_consensus.cc:2781] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:27.964928 25673 raft_consensus.cc:482] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:27.965050 25673 raft_consensus.cc:3037] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:27.967047 25673 raft_consensus.cc:504] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.967689 25673 leader_election.cc:288] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 election: Requested vote from peers 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:27.968181 25601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99331be41540c28a5cd2fb6992f825" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f"
I20260926 07:33:27.968184 25318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99331be41540c28a5cd2fb6992f825" candidate_uuid: "6ef65d6eb86d486a88fa245e76b093a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf9278c33c046859c1ea531f8101487"
I20260926 07:33:27.968441 25601 raft_consensus.cc:3037] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:27.968441 25318 raft_consensus.cc:3037] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:27.971486 25318 raft_consensus.cc:2454] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 1.
I20260926 07:33:27.971472 25601 raft_consensus.cc:2454] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef65d6eb86d486a88fa245e76b093a6 in term 1.
I20260926 07:33:27.971993 25391 leader_election.cc:302] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef65d6eb86d486a88fa245e76b093a6, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:27.972313 25673 raft_consensus.cc:2781] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:27.972465 25673 raft_consensus.cc:686] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Running, Role: LEADER
I20260926 07:33:27.972783 25673 consensus_queue.cc:237] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:27.975329 25160 catalog_manager.cc:5495] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66). New cstate: current_term: 1 leader_uuid: "6ef65d6eb86d486a88fa245e76b093a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260926 07:33:28.006116 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:28.000387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.22.221.66:41739)
0926 07:33:28.000943 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:28.000950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:28.000973 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:33:28.001050 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:28.001055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:28.001128 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:28.001236 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:28.001643 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.002415 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.003792 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.004455 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.004733 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:28.005338 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:28.005456 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:28.005559 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:28.005715 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:33:28.005793 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260926 07:33:28.019016 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:28.014286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:33067 (local address 127.22.221.66:41739)
0926 07:33:28.014438 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:28.014442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:28.014828 (+ 386us) server_negotiation.cc:407] Connection header received
0926 07:33:28.014967 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:28.014970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:28.015028 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:28.015100 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:28.015516 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.016332 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.017598 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.018029 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.018122 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:28.018436 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:28.018527 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:28.018591 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:28.018713 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 07:33:28.018771 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
I20260926 07:33:28.019109 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:28.014136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41739 (local address 127.22.221.66:33067)
0926 07:33:28.014748 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:28.014763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:28.014875 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:28.015126 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:28.015129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:28.015137 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:33:28.015405 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:28.015412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.016442 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.016445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:28.017438 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:28.017449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.018170 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.018174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:28.018282 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:28.018314 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:33:28.018609 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:33:28.018617 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:33:28.018677 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:33:28.018775 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":14,"thread_start_us":334,"threads_started":1}
I20260926 07:33:28.027064 25318 raft_consensus.cc:1261] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Refusing update from remote peer 6ef65d6eb86d486a88fa245e76b093a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:28.027036 25601 raft_consensus.cc:1261] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Refusing update from remote peer 6ef65d6eb86d486a88fa245e76b093a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:28.027573 25673 consensus_queue.cc:1035] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:28.027864 25735 consensus_queue.cc:1035] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:28.079947 25161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:28.080204 25161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:28.083158 25161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Sending DeleteTablet for 3 replicas of tablet 1a99331be41540c28a5cd2fb6992f825
I20260926 07:33:28.084210 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99331be41540c28a5cd2fb6992f825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:28 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49520
I20260926 07:33:28.084254 25575 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99331be41540c28a5cd2fb6992f825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:28 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49126
I20260926 07:33:28.084823 25751 tablet_replica.cc:330] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: stopping tablet replica
I20260926 07:33:28.084966 25749 tablet_replica.cc:330] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: stopping tablet replica
I20260926 07:33:28.085064 25751 raft_consensus.cc:2229] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:28.085088 25436 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99331be41540c28a5cd2fb6992f825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:28 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49136
I20260926 07:33:28.085177 25749 raft_consensus.cc:2229] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:28.085275 25751 raft_consensus.cc:2258] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:28.085352 25750 tablet_replica.cc:330] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: stopping tablet replica
I20260926 07:33:28.085368 25749 raft_consensus.cc:2258] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:28.085549 25750 raft_consensus.cc:2229] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:28.086477 25749 ts_tablet_manager.cc:1905] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:28.086685 25161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:33:28.086266 25750 raft_consensus.cc:2258] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:33:28.087430 25161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:28.088318 25750 ts_tablet_manager.cc:1905] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:28.090158 25751 ts_tablet_manager.cc:1905] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:28.094724 25749 ts_tablet_manager.cc:1918] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:28.094873 25749 log.cc:1196] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/1a99331be41540c28a5cd2fb6992f825
I20260926 07:33:28.095459 25750 ts_tablet_manager.cc:1918] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:28.095621 25750 log.cc:1196] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/1a99331be41540c28a5cd2fb6992f825
I20260926 07:33:28.095824 25749 ts_tablet_manager.cc:1939] T 1a99331be41540c28a5cd2fb6992f825 P 2bf9278c33c046859c1ea531f8101487: Deleting consensus metadata
I20260926 07:33:28.096073 25750 ts_tablet_manager.cc:1939] T 1a99331be41540c28a5cd2fb6992f825 P 6ef65d6eb86d486a88fa245e76b093a6: Deleting consensus metadata
I20260926 07:33:28.096954 25146 catalog_manager.cc:4841] TS 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901): tablet 1a99331be41540c28a5cd2fb6992f825 (table test-table [id=4a6077c8b3e04877bdb41a7a5fc9682e]) successfully deleted
I20260926 07:33:28.097196 25146 catalog_manager.cc:4841] TS 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739): tablet 1a99331be41540c28a5cd2fb6992f825 (table test-table [id=4a6077c8b3e04877bdb41a7a5fc9682e]) successfully deleted
I20260926 07:33:28.099470 25751 ts_tablet_manager.cc:1918] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:33:28.099647 25751 log.cc:1196] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/1a99331be41540c28a5cd2fb6992f825
I20260926 07:33:28.100106 25751 ts_tablet_manager.cc:1939] T 1a99331be41540c28a5cd2fb6992f825 P d022a75c4d94425ca1e1aaba4787593f: Deleting consensus metadata
I20260926 07:33:28.100317 25297 tablet_service.cc:1467] Processing CreateTablet for tablet b87c357cad4f4033a01b0407e99bf66d (DEFAULT_TABLE table=test-table [id=d24dd529ed5c4a32bd3e6c2c77e26465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:28.100744 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87c357cad4f4033a01b0407e99bf66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:28.100986 25575 tablet_service.cc:1467] Processing CreateTablet for tablet b87c357cad4f4033a01b0407e99bf66d (DEFAULT_TABLE table=test-table [id=d24dd529ed5c4a32bd3e6c2c77e26465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:28.101393 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87c357cad4f4033a01b0407e99bf66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:28.101508 25148 catalog_manager.cc:4841] TS d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809): tablet 1a99331be41540c28a5cd2fb6992f825 (table test-table [id=4a6077c8b3e04877bdb41a7a5fc9682e]) successfully deleted
I20260926 07:33:28.103101 25436 tablet_service.cc:1467] Processing CreateTablet for tablet b87c357cad4f4033a01b0407e99bf66d (DEFAULT_TABLE table=test-table [id=d24dd529ed5c4a32bd3e6c2c77e26465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:28.103549 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87c357cad4f4033a01b0407e99bf66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:28.107219 25761 tablet_bootstrap.cc:492] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Bootstrap starting.
I20260926 07:33:28.107565 25760 tablet_bootstrap.cc:492] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Bootstrap starting.
I20260926 07:33:28.109016 25761 tablet_bootstrap.cc:654] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:28.109215 25760 tablet_bootstrap.cc:654] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:28.110872 25762 tablet_bootstrap.cc:492] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Bootstrap starting.
I20260926 07:33:28.113125 25762 tablet_bootstrap.cc:654] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:28.113631 25760 tablet_bootstrap.cc:492] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: No bootstrap required, opened a new log
I20260926 07:33:28.113785 25760 ts_tablet_manager.cc:1397] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260926 07:33:28.114504 25760 raft_consensus.cc:348] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.114775 25760 raft_consensus.cc:374] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:28.115054 25760 raft_consensus.cc:729] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d022a75c4d94425ca1e1aaba4787593f, State: Initialized, Role: FOLLOWER
I20260926 07:33:28.115411 25760 consensus_queue.cc:260] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.116160 25760 ts_tablet_manager.cc:1428] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:33:28.118472 25761 tablet_bootstrap.cc:492] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: No bootstrap required, opened a new log
I20260926 07:33:28.118713 25761 ts_tablet_manager.cc:1397] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260926 07:33:28.119578 25761 raft_consensus.cc:348] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.119932 25761 raft_consensus.cc:374] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:28.120050 25761 raft_consensus.cc:729] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Initialized, Role: FOLLOWER
I20260926 07:33:28.120496 25761 consensus_queue.cc:260] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.121167 25761 ts_tablet_manager.cc:1428] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:28.123375 25762 tablet_bootstrap.cc:492] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: No bootstrap required, opened a new log
I20260926 07:33:28.123569 25762 ts_tablet_manager.cc:1397] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260926 07:33:28.124408 25762 raft_consensus.cc:348] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.124680 25762 raft_consensus.cc:374] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:28.124763 25762 raft_consensus.cc:729] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef65d6eb86d486a88fa245e76b093a6, State: Initialized, Role: FOLLOWER
I20260926 07:33:28.125049 25762 consensus_queue.cc:260] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.125795 25762 ts_tablet_manager.cc:1428] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:33:28.170491 25740 raft_consensus.cc:482] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:28.170740 25740 raft_consensus.cc:504] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.171658 25740 leader_election.cc:288] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:28.172189 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87c357cad4f4033a01b0407e99bf66d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f" is_pre_election: true
I20260926 07:33:28.172215 25457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87c357cad4f4033a01b0407e99bf66d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef65d6eb86d486a88fa245e76b093a6" is_pre_election: true
I20260926 07:33:28.172457 25600 raft_consensus.cc:2454] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 0.
I20260926 07:33:28.172508 25457 raft_consensus.cc:2454] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 0.
I20260926 07:33:28.172941 25251 leader_election.cc:302] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:28.173241 25740 raft_consensus.cc:2781] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:28.173385 25740 raft_consensus.cc:482] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:28.173524 25740 raft_consensus.cc:3037] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:28.175993 25740 raft_consensus.cc:504] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.176573 25740 leader_election.cc:288] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 election: Requested vote from peers 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739), d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809)
I20260926 07:33:28.177080 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87c357cad4f4033a01b0407e99bf66d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d022a75c4d94425ca1e1aaba4787593f"
I20260926 07:33:28.177065 25457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87c357cad4f4033a01b0407e99bf66d" candidate_uuid: "2bf9278c33c046859c1ea531f8101487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef65d6eb86d486a88fa245e76b093a6"
I20260926 07:33:28.177246 25600 raft_consensus.cc:3037] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:28.177332 25457 raft_consensus.cc:3037] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:28.179347 25600 raft_consensus.cc:2454] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 1.
I20260926 07:33:28.179718 25251 leader_election.cc:302] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9278c33c046859c1ea531f8101487, d022a75c4d94425ca1e1aaba4787593f; no voters:
I20260926 07:33:28.179978 25740 raft_consensus.cc:2781] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:28.180135 25740 raft_consensus.cc:686] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf9278c33c046859c1ea531f8101487, State: Running, Role: LEADER
I20260926 07:33:28.180414 25740 consensus_queue.cc:237] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } }
I20260926 07:33:28.180884 25457 raft_consensus.cc:2454] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf9278c33c046859c1ea531f8101487 in term 1.
I20260926 07:33:28.186748 25160 catalog_manager.cc:5495] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bf9278c33c046859c1ea531f8101487 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "2bf9278c33c046859c1ea531f8101487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf9278c33c046859c1ea531f8101487" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260926 07:33:28.214522 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:28.204915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.22.221.65:45901)
0926 07:33:28.208678 (+ 3763us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:28.208682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:28.208702 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:33:28.208765 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:28.208768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:28.208829 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:28.208917 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:28.209388 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.210285 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.212553 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:28.212971 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:28.213074 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:28.213632 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:28.213736 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:28.213806 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:28.214026 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 07:33:28.214107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3643,"thread_start_us":3503,"threads_started":1}
I20260926 07:33:28.224256 25600 raft_consensus.cc:1261] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Refusing update from remote peer 2bf9278c33c046859c1ea531f8101487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:28.224308 25457 raft_consensus.cc:1261] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Refusing update from remote peer 2bf9278c33c046859c1ea531f8101487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:28.224921 25744 consensus_queue.cc:1035] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d022a75c4d94425ca1e1aaba4787593f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:28.225266 25740 consensus_queue.cc:1035] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef65d6eb86d486a88fa245e76b093a6" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:28.234402 25732 mvcc.cc:205] Tried to move back new op lower bound from 7333511201681190912 to 7333511201509462016. Current Snapshot: MvccSnapshot[applied={T|T < 7333511201681190912}]
W20260926 07:33:28.251926 25732 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:33:29.837285 25161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:29.837586 25161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:29.841836 25161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dff99337bf64cb6a4476f1f4016762c: Sending DeleteTablet for 3 replicas of tablet b87c357cad4f4033a01b0407e99bf66d
I20260926 07:33:29.842646 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet b87c357cad4f4033a01b0407e99bf66d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:29 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49520
I20260926 07:33:29.842741 25575 tablet_service.cc:1514] Processing DeleteTablet for tablet b87c357cad4f4033a01b0407e99bf66d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:29 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49126
I20260926 07:33:29.842742 25436 tablet_service.cc:1514] Processing DeleteTablet for tablet b87c357cad4f4033a01b0407e99bf66d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:29 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49136
I20260926 07:33:29.843376 25808 tablet_replica.cc:330] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: stopping tablet replica
I20260926 07:33:29.843647 25808 raft_consensus.cc:2229] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:29.843720 25810 tablet_replica.cc:330] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: stopping tablet replica
I20260926 07:33:29.843837 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25232
I20260926 07:33:29.844026 25810 raft_consensus.cc:2229] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:29.844141 25808 raft_consensus.cc:2258] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:29.844307 25810 raft_consensus.cc:2258] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:29.844884 25809 tablet_replica.cc:330] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: stopping tablet replica
I20260926 07:33:29.845063 25809 raft_consensus.cc:2229] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:29.845299 25809 raft_consensus.cc:2258] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:29.846696 25809 ts_tablet_manager.cc:1905] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:29.847402 25810 ts_tablet_manager.cc:1905] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:29.851544 25808 ts_tablet_manager.cc:1905] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:29.853504 25809 ts_tablet_manager.cc:1918] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:33:29.853722 25809 log.cc:1196] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/b87c357cad4f4033a01b0407e99bf66d
I20260926 07:33:29.853997 25810 ts_tablet_manager.cc:1918] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:33:29.854195 25810 log.cc:1196] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/b87c357cad4f4033a01b0407e99bf66d
I20260926 07:33:29.854279 25809 ts_tablet_manager.cc:1939] T b87c357cad4f4033a01b0407e99bf66d P d022a75c4d94425ca1e1aaba4787593f: Deleting consensus metadata
I20260926 07:33:29.854697 25810 ts_tablet_manager.cc:1939] T b87c357cad4f4033a01b0407e99bf66d P 6ef65d6eb86d486a88fa245e76b093a6: Deleting consensus metadata
I20260926 07:33:29.855581 25148 catalog_manager.cc:4841] TS d022a75c4d94425ca1e1aaba4787593f (127.22.221.67:46809): tablet b87c357cad4f4033a01b0407e99bf66d (table test-table [id=d24dd529ed5c4a32bd3e6c2c77e26465]) successfully deleted
I20260926 07:33:29.855774 25146 catalog_manager.cc:4841] TS 6ef65d6eb86d486a88fa245e76b093a6 (127.22.221.66:41739): tablet b87c357cad4f4033a01b0407e99bf66d (table test-table [id=d24dd529ed5c4a32bd3e6c2c77e26465]) successfully deleted
I20260926 07:33:29.860613 25808 ts_tablet_manager.cc:1918] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:33:29.860729 25808 log.cc:1196] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/b87c357cad4f4033a01b0407e99bf66d
I20260926 07:33:29.861212 25808 ts_tablet_manager.cc:1939] T b87c357cad4f4033a01b0407e99bf66d P 2bf9278c33c046859c1ea531f8101487: Deleting consensus metadata
I20260926 07:33:29.862203 25146 catalog_manager.cc:4841] TS 2bf9278c33c046859c1ea531f8101487 (127.22.221.65:45901): tablet b87c357cad4f4033a01b0407e99bf66d (table test-table [id=d24dd529ed5c4a32bd3e6c2c77e26465]) successfully deleted
Sep 26 07:33:29 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:29.875844 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:29.845050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.22.221.65:45901)
0926 07:33:29.856205 (+ 11155us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:29.856212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:29.856564 (+ 352us) server_negotiation.cc:407] Connection header received
0926 07:33:29.856759 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:29.856766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:29.856860 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:29.856968 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:29.858271 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:29.859141 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:29.860484 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:29.861374 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:29.861515 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:29.869767 (+ 8252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:29.869782 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:29.869798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:29.869833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:29.872915 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:29.873663 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:29.873670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:29.873673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:29.873816 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:29.874213 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:29.874218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:29.874221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:29.874566 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:29.874889 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:29.875175 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 07:33:29.875461 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10947,"thread_start_us":10899,"threads_started":1}
W20260926 07:33:29.876477 25363 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49550
W20260926 07:33:29.877166 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25232: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:29.877389 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 25232
I20260926 07:33:29.900156 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25372
Sep 26 07:33:29 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:29.915692 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:29.900863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.22.221.66:41739)
0926 07:33:29.901378 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:29.901383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:29.901398 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:33:29.901451 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:29.901455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:29.901518 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:29.901638 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:29.902899 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:29.903728 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:29.905001 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:29.905396 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:29.905533 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:29.910102 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:29.910116 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:29.910132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:29.910169 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:29.913221 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:29.913848 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:29.913856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:29.913858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:29.913986 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:29.914288 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:29.914291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:29.914293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:29.914631 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:29.914945 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:29.915134 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:33:29.915373 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
W20260926 07:33:29.916332 25502 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49152
W20260926 07:33:29.916923 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25372: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:29.917061 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 25372
I20260926 07:33:29.942672 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25511
Sep 26 07:33:29 dist-test-slave-qlq8 krb5kdc[25116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:29.968973 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:29.943609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.22.221.67:46809)
0926 07:33:29.951766 (+ 8157us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:29.951782 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:29.951803 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:33:29.951858 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:29.951862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:29.951928 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:29.952048 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:29.953887 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:29.954852 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:29.956760 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:29.957200 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:29.957389 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:29.962684 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:29.962698 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:29.962715 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:29.962759 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:29.966107 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:29.966856 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:29.966863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:29.966865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:29.966990 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:29.967434 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:29.967439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:29.967442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:29.967822 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:29.968138 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:29.968430 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 07:33:29.968652 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8009,"thread_start_us":7844,"threads_started":1}
W20260926 07:33:29.969780 25641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49142
W20260926 07:33:29.970399 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25511: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:29.970556 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 25511
I20260926 07:33:29.988483 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25129
I20260926 07:33:29.989728 25201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:30.106554 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 25129
2026-09-26T07:33:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8519 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:30 dist-test-slave-qlq8 krb5kdc[25818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:30 dist-test-slave-qlq8 krb5kdc[25818](info): set up 2 sockets
Sep 26 07:33:30 dist-test-slave-qlq8 krb5kdc[25818](info): commencing operation
krb5kdc: starting...
W20260926 07:33:31.191562 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:31 dist-test-slave-qlq8 krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:31.387024 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:32893
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:31.706746 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:31.707118 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:31.707304 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:31.714619 25831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:31.714763 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:31.714874 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:31.714984 25831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:31.715077 25831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:31.724975 25831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:31.726393 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:31.727801 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:31.740217 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:31.740252 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:31.740188 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:31.741652 25831 server_base.cc:1034] running on GCE node
I20260926 07:33:31.742429 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:31.744195 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:31.745433 25831 hybrid_clock.cc:648] HybridClock initialized: now 1790408011745364 us; error 66 us; skew 500 ppm
I20260926 07:33:31.745775 25831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:31 dist-test-slave-qlq8 krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408011, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:31.749166 25831 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:31.750907 25831 webserver.cc:460] Webserver started at http://127.22.221.126:35833/ using document root <none> and password file <none>
I20260926 07:33:31.751757 25831 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:31.751873 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:31.752132 25831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:31.755403 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "ff1ad2d16857499ca088c8c244fdc778"
format_stamp: "Formatted at 2026-09-26 07:33:31 on dist-test-slave-qlq8"
server_key: "76465eec41aad5bdcb29dae3d529615e"
server_key_iv: "8274af63b3b1088ff78371b107425c3b"
server_key_version: "encryptionkey@0"
I20260926 07:33:31.756175 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "ff1ad2d16857499ca088c8c244fdc778"
format_stamp: "Formatted at 2026-09-26 07:33:31 on dist-test-slave-qlq8"
server_key: "76465eec41aad5bdcb29dae3d529615e"
server_key_iv: "8274af63b3b1088ff78371b107425c3b"
server_key_version: "encryptionkey@0"
I20260926 07:33:31.761665 25831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:33:31.765293 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:31.766656 25831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:31.766858 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "ff1ad2d16857499ca088c8c244fdc778"
format_stamp: "Formatted at 2026-09-26 07:33:31 on dist-test-slave-qlq8"
server_key: "76465eec41aad5bdcb29dae3d529615e"
server_key_iv: "8274af63b3b1088ff78371b107425c3b"
server_key_version: "encryptionkey@0"
I20260926 07:33:31.767067 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:31.783833 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:31.788343 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:31.788643 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:31.807367 25831 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:32893
I20260926 07:33:31.807418 25897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:32893 every 8 connection(s)
I20260926 07:33:31.808729 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:31.813316 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:31.815301 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 25831
I20260926 07:33:31.815551 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:31.816159 23413 external_mini_cluster.cc:1428] Setting key 5c6c74c66b80ff97e103f0c9ff034b74
I20260926 07:33:31.829422 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778: Bootstrap starting.
Sep 26 07:33:31 dist-test-slave-qlq8 krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:31.833120 25898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:31.834666 25898 log.cc:824] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:31.838306 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778: No bootstrap required, opened a new log
I20260926 07:33:31.842079 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:31.818217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.22.221.126:32893)
0926 07:33:31.819092 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:31.819112 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:31.819178 (+ 66us) server_negotiation.cc:407] Connection header received
0926 07:33:31.821152 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:31.821183 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:31.821772 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:31.822224 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:31.823636 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:31.825938 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:31.827286 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:31.828438 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:31.828642 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:31.832738 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:31.832749 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:31.832772 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:31.832828 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:31.837131 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:31.837842 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:31.837852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:31.837860 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:31.838085 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:31.838492 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:31.838497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:31.838500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:31.839336 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:31.839693 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:31.839989 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:33:31.840477 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":349,"threads_started":1}
I20260926 07:33:31.847834 25898 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ad2d16857499ca088c8c244fdc778" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 32893 } }
I20260926 07:33:31.848155 25898 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:31.848239 25898 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1ad2d16857499ca088c8c244fdc778, State: Initialized, Role: FOLLOWER
I20260926 07:33:31.848845 25898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ad2d16857499ca088c8c244fdc778" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 32893 } }
I20260926 07:33:31.849113 25898 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:31.849313 25898 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:31.849484 25898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:31.851675 25898 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ad2d16857499ca088c8c244fdc778" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 32893 } }
I20260926 07:33:31.852169 25898 leader_election.cc:302] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff1ad2d16857499ca088c8c244fdc778; no voters:
I20260926 07:33:31.852751 25898 leader_election.cc:288] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:31.852941 25903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:31.853614 25903 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [term 1 LEADER]: Becoming Leader. State: Replica: ff1ad2d16857499ca088c8c244fdc778, State: Running, Role: LEADER
I20260926 07:33:31.854132 25903 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ad2d16857499ca088c8c244fdc778" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 32893 } }
I20260926 07:33:31.854781 25898 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:31.856760 25904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff1ad2d16857499ca088c8c244fdc778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ad2d16857499ca088c8c244fdc778" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 32893 } } }
I20260926 07:33:31.857164 25904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:31.858076 25908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:31.857921 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1ad2d16857499ca088c8c244fdc778. Latest consensus state: current_term: 1 leader_uuid: "ff1ad2d16857499ca088c8c244fdc778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ad2d16857499ca088c8c244fdc778" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 32893 } } }
I20260926 07:33:31.858237 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:31.863719 25908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:31.873775 25908 catalog_manager.cc:1347] Generated new cluster ID: a2ea12f8e2d54af191a4bf9d5ec5c828
I20260926 07:33:31.873905 25908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:31.885973 25908 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:31.887908 25908 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:31.901986 25908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff1ad2d16857499ca088c8c244fdc778: Generated new TSK 0
I20260926 07:33:31.902962 25908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:31.963120 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:32893
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:32.297215 25924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:32.297524 25924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:32.297694 25924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:32.305562 25924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:32.305687 25924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:32.305888 25924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:32.316285 25924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:32893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:32.317637 25924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:32.319070 25924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:32.332698 25932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:32.332790 25930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:32.332736 25929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:32.337675 25924 server_base.cc:1034] running on GCE node
I20260926 07:33:32.338333 25924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:32.339387 25924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:32.340646 25924 hybrid_clock.cc:648] HybridClock initialized: now 1790408012340584 us; error 57 us; skew 500 ppm
I20260926 07:33:32.341102 25924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:32 dist-test-slave-qlq8 krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:32.344357 25924 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:32.345909 25924 webserver.cc:460] Webserver started at http://127.22.221.65:38983/ using document root <none> and password file <none>
I20260926 07:33:32.346648 25924 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:32.346798 25924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:32.347111 25924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:32.350057 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "f2e76e69852947d196f411662d5cbb79"
format_stamp: "Formatted at 2026-09-26 07:33:32 on dist-test-slave-qlq8"
server_key: "5b38c70f83d30f82318bb647830cd95c"
server_key_iv: "93c7dad0ababb063c90821c1a5940c6f"
server_key_version: "encryptionkey@0"
I20260926 07:33:32.350920 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e76e69852947d196f411662d5cbb79"
format_stamp: "Formatted at 2026-09-26 07:33:32 on dist-test-slave-qlq8"
server_key: "5b38c70f83d30f82318bb647830cd95c"
server_key_iv: "93c7dad0ababb063c90821c1a5940c6f"
server_key_version: "encryptionkey@0"
I20260926 07:33:32.356343 25924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:33:32.359892 25939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:32.361282 25924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:33:32.361460 25924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "f2e76e69852947d196f411662d5cbb79"
format_stamp: "Formatted at 2026-09-26 07:33:32 on dist-test-slave-qlq8"
server_key: "5b38c70f83d30f82318bb647830cd95c"
server_key_iv: "93c7dad0ababb063c90821c1a5940c6f"
server_key_version: "encryptionkey@0"
I20260926 07:33:32.361668 25924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:32.403874 25924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:32.408100 25924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:32.408411 25924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:32.409374 25924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:32.410753 25924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:32.410859 25924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:32.411010 25924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:32.411084 25924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:32.443641 25924 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:42429
I20260926 07:33:32.443701 26051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:42429 every 8 connection(s)
I20260926 07:33:32.444918 25924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:32.453154 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 25924
I20260926 07:33:32.453392 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:32.454075 23413 external_mini_cluster.cc:1428] Setting key 7112ed25a9f925a81ba19c6da926f376
Sep 26 07:33:32 dist-test-slave-qlq8 krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:32.467038 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:32.447500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:33197 (local address 127.22.221.126:32893)
0926 07:33:32.447921 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:32.447926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:32.448897 (+ 971us) server_negotiation.cc:407] Connection header received
0926 07:33:32.450785 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:32.450789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:32.450856 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:32.450953 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:32.452452 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.453169 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.455192 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.455676 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.455847 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:32.459401 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:32.459406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:32.459409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:32.459440 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:32.462420 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:32.463568 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:32.463574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:32.463576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:32.463772 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:32.464376 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:32.464382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:32.464385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:32.464835 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:32.464986 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:32.466180 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0926 07:33:32.466581 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":203,"threads_started":1}
I20260926 07:33:32.467962 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:32.448063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:32893 (local address 127.22.221.65:33197)
0926 07:33:32.448733 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:32.448775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:32.450528 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:32.451158 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:32.451166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:32.451738 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:32.452291 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:32.452308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.453405 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.453409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:32.455058 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:32.455073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.455940 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.455949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:32.456309 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:32.457002 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:32.457025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:32.459143 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:32.462581 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:32.462593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:32.462612 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:32.463383 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:32.463907 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:32.463912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:32.463916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:32.464238 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:32.464990 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:32.464999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:32.465599 (+ 600us) client_negotiation.cc:769] Sending connection context
0926 07:33:32.465971 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 07:33:32.466544 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":262,"threads_started":1}
I20260926 07:33:32.472076 26052 heartbeater.cc:347] Connected to a master server at 127.22.221.126:32893
I20260926 07:33:32.472445 26052 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:32.473214 26052 heartbeater.cc:511] Master 127.22.221.126:32893 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:32.475545 25863 ts_manager.cc:194] Registered new tserver with Master: f2e76e69852947d196f411662d5cbb79 (127.22.221.65:42429)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:33:32.477236 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:33197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:32.502254 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:32893
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:32.836908 26058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:32.837265 26058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:32.837438 26058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:32.844942 26058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:32.845112 26058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:32.845294 26058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:32.855168 26058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:32893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:32.856647 26058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:32.857954 26058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:32.871672 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:32.871857 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:32.871884 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:32.872776 26058 server_base.cc:1034] running on GCE node
I20260926 07:33:32.873401 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:32.874399 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:32.875639 26058 hybrid_clock.cc:648] HybridClock initialized: now 1790408012875593 us; error 62 us; skew 500 ppm
I20260926 07:33:32.875957 26058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:32 dist-test-slave-qlq8 krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:32.879159 26058 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:32.880712 26058 webserver.cc:460] Webserver started at http://127.22.221.66:40271/ using document root <none> and password file <none>
I20260926 07:33:32.881408 26058 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:32.881556 26058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:32.881870 26058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:32.884912 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "0ca21f631d4e4a52a119bbc1dd23ced2"
format_stamp: "Formatted at 2026-09-26 07:33:32 on dist-test-slave-qlq8"
server_key: "1f7724b4bf962294fde1ecd92a0e1340"
server_key_iv: "2c8f7485f64e3160263cbb0d473cef40"
server_key_version: "encryptionkey@0"
I20260926 07:33:32.885739 26058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "0ca21f631d4e4a52a119bbc1dd23ced2"
format_stamp: "Formatted at 2026-09-26 07:33:32 on dist-test-slave-qlq8"
server_key: "1f7724b4bf962294fde1ecd92a0e1340"
server_key_iv: "2c8f7485f64e3160263cbb0d473cef40"
server_key_version: "encryptionkey@0"
I20260926 07:33:32.891113 26058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:33:32.894639 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:32.896111 26058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:32.896291 26058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "0ca21f631d4e4a52a119bbc1dd23ced2"
format_stamp: "Formatted at 2026-09-26 07:33:32 on dist-test-slave-qlq8"
server_key: "1f7724b4bf962294fde1ecd92a0e1340"
server_key_iv: "2c8f7485f64e3160263cbb0d473cef40"
server_key_version: "encryptionkey@0"
I20260926 07:33:32.896502 26058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:32.919561 26058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:32.923653 26058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:32.923974 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:32.924952 26058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:32.926304 26058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:32.926415 26058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:32.926573 26058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:32.926649 26058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:32.960952 26058 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:45379
I20260926 07:33:32.961068 26185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:45379 every 8 connection(s)
I20260926 07:33:32.962329 26058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:32.971460 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26058
I20260926 07:33:32.971679 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:32.972133 23413 external_mini_cluster.cc:1428] Setting key 355d0e9e95bc08bed7cbc6f30024396a
Sep 26 07:33:32 dist-test-slave-qlq8 krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408012, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:32.983847 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:32.964954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:39831 (local address 127.22.221.126:32893)
0926 07:33:32.965162 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:32.965168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:32.966555 (+ 1387us) server_negotiation.cc:407] Connection header received
0926 07:33:32.968464 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:32.968468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:32.968540 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:32.968630 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:32.970110 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.970891 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.972830 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.973461 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.973596 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:32.977291 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:32.977300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:32.977308 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:32.977357 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:32.980616 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:32.981576 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:32.981580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:32.981582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:32.981705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:32.982171 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:32.982174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:32.982176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:32.982431 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:32.982537 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:32.983422 (+ 885us) server_negotiation.cc:299] Negotiation successful
0926 07:33:32.983602 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:33:32.985383 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:32.965582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:32893 (local address 127.22.221.66:39831)
0926 07:33:32.966410 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:32.966450 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:32.968194 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:32.968837 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:32.968844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:32.969365 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:32.969933 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:32.969952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.971050 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.971054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:32.972651 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:32.972664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:32.973659 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:32.973664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:32.973883 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:32.974630 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:32.974651 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:32.977042 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:32.980767 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:32.980780 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:32.980798 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:32.981451 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:32.981810 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:32.981817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:32.981824 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:32.982059 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:32.982551 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:32.982562 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:32.982994 (+ 432us) client_negotiation.cc:769] Sending connection context
0926 07:33:32.983316 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 07:33:32.983886 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":321,"threads_started":1}
I20260926 07:33:32.986841 26186 heartbeater.cc:347] Connected to a master server at 127.22.221.126:32893
I20260926 07:33:32.987159 26186 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:32.987900 26186 heartbeater.cc:511] Master 127.22.221.126:32893 requested a full tablet report, sending...
I20260926 07:33:32.989346 25863 ts_manager.cc:194] Registered new tserver with Master: 0ca21f631d4e4a52a119bbc1dd23ced2 (127.22.221.66:45379)
I20260926 07:33:32.990165 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:39831
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:33.019075 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:32893
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:33.349393 26191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:33.349786 26191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:33.349948 26191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:33.357494 26191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:33.357648 26191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:33.357827 26191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:33.368088 26191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:32893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:33.369457 26191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:33.370889 26191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:33.385007 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:33.384920 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:33.385012 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:33.386475 26191 server_base.cc:1034] running on GCE node
I20260926 07:33:33.387086 26191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:33.388151 26191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:33.389396 26191 hybrid_clock.cc:648] HybridClock initialized: now 1790408013389346 us; error 41 us; skew 500 ppm
I20260926 07:33:33.389732 26191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:33 dist-test-slave-qlq8 krb5kdc[25818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408013, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:33.393025 26191 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:33.394686 26191 webserver.cc:460] Webserver started at http://127.22.221.67:34007/ using document root <none> and password file <none>
I20260926 07:33:33.395442 26191 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:33.395550 26191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:33.395820 26191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:33.398823 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "0865831fe52a4e3fb57a3584f49dc2fd"
format_stamp: "Formatted at 2026-09-26 07:33:33 on dist-test-slave-qlq8"
server_key: "c7ea3a12a4b19052b26385d71701aa01"
server_key_iv: "5890bbb9e59a621b6b8f8bccbb6ecb28"
server_key_version: "encryptionkey@0"
I20260926 07:33:33.399699 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "0865831fe52a4e3fb57a3584f49dc2fd"
format_stamp: "Formatted at 2026-09-26 07:33:33 on dist-test-slave-qlq8"
server_key: "c7ea3a12a4b19052b26385d71701aa01"
server_key_iv: "5890bbb9e59a621b6b8f8bccbb6ecb28"
server_key_version: "encryptionkey@0"
I20260926 07:33:33.405051 26191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:33:33.408699 26206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:33.410090 26191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:33.410277 26191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "0865831fe52a4e3fb57a3584f49dc2fd"
format_stamp: "Formatted at 2026-09-26 07:33:33 on dist-test-slave-qlq8"
server_key: "c7ea3a12a4b19052b26385d71701aa01"
server_key_iv: "5890bbb9e59a621b6b8f8bccbb6ecb28"
server_key_version: "encryptionkey@0"
I20260926 07:33:33.410487 26191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:33.435070 26191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:33.439466 26191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:33.439770 26191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:33.440795 26191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:33.442157 26191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:33.442239 26191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:33.442407 26191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:33.442461 26191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:33.477028 26191 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:38547
I20260926 07:33:33.477085 26318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:38547 every 8 connection(s)
I20260926 07:33:33.478753 26191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:33.481045 26052 heartbeater.cc:503] Master 127.22.221.126:32893 was elected leader, sending a full tablet report...
I20260926 07:33:33.488531 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26191
I20260926 07:33:33.488759 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:33.489302 23413 external_mini_cluster.cc:1428] Setting key edc010388e9bba789849affd3d2b802b
Sep 26 07:33:33 dist-test-slave-qlq8 krb5kdc[25818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408013, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:33.498986 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:33.481359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:33521 (local address 127.22.221.126:32893)
0926 07:33:33.481716 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:33.481721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:33.482966 (+ 1245us) server_negotiation.cc:407] Connection header received
0926 07:33:33.484835 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:33.484839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:33.484901 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:33.484987 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:33.486570 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:33.487406 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:33.488988 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:33.489630 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:33.489772 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:33.493390 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:33.493395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:33.493399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:33.493429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:33.495916 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:33.496933 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:33.496937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:33.496940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:33.497065 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:33.497520 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:33.497523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:33.497525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:33.497781 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:33.497897 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:33.498573 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 07:33:33.498734 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260926 07:33:33.499998 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:33.481994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:32893 (local address 127.22.221.67:33521)
0926 07:33:33.482829 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:33.482875 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:33.484588 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:33.485192 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:33.485200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:33.485699 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:33.486362 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:33.486392 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:33.487535 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:33.487539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:33.488809 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:33.488821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:33.489837 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:33.489842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:33.490167 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:33.490913 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:33.490936 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:33.493200 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:33.496045 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:33.496052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:33.496064 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:33.496803 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:33.497197 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:33.497200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:33.497203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:33.497415 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:33.497896 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:33.497901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:33.498231 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 07:33:33.498483 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 07:33:33.498871 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":361,"threads_started":1}
I20260926 07:33:33.501561 26319 heartbeater.cc:347] Connected to a master server at 127.22.221.126:32893
I20260926 07:33:33.501837 26319 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:33.502540 26319 heartbeater.cc:511] Master 127.22.221.126:32893 requested a full tablet report, sending...
I20260926 07:33:33.504284 25863 ts_manager.cc:194] Registered new tserver with Master: 0865831fe52a4e3fb57a3584f49dc2fd (127.22.221.67:38547)
I20260926 07:33:33.504931 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:33.505427 25863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:33521
W20260926 07:33:33.536113 26054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:33.533625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.22.221.126:32893)
0926 07:33:33.533863 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:33.533872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:33.534244 (+ 372us) server_negotiation.cc:407] Connection header received
0926 07:33:33.534396 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:33.534401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:33.534467 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:33.534555 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:33.535840 (+ 1285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35246: BlockingRecv error: recv got EOF from 127.0.0.1:35246 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:33:33.539165 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25924
W20260926 07:33:33.542116 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25924: Not authorized: Client connection negotiation failed: client connection to 127.22.221.65:42429: server requires authentication, but client does not have Kerberos credentials available
I20260926 07:33:33.542323 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 25924
I20260926 07:33:33.554987 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26058
W20260926 07:33:33.557586 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26058: Not authorized: Client connection negotiation failed: client connection to 127.22.221.66:45379: server requires authentication, but client does not have Kerberos credentials available
I20260926 07:33:33.557739 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26058
I20260926 07:33:33.573539 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26191
W20260926 07:33:33.576643 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26191: Not authorized: Client connection negotiation failed: client connection to 127.22.221.67:38547: server requires authentication, but client does not have Kerberos credentials available
I20260926 07:33:33.576809 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26191
I20260926 07:33:33.591661 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 25831
I20260926 07:33:33.592933 25893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:33.695690 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 25831
2026-09-26T07:33:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3586 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:33 dist-test-slave-qlq8 krb5kdc[26336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:33 dist-test-slave-qlq8 krb5kdc[26336](info): set up 2 sockets
Sep 26 07:33:33 dist-test-slave-qlq8 krb5kdc[26336](info): commencing operation
krb5kdc: starting...
W20260926 07:33:34.787643 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:34 dist-test-slave-qlq8 krb5kdc[26336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:34.987612 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:38171
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:35.307739 26349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:35.308086 26349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:35.308266 26349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:35.315532 26349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:35.315670 26349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:35.315789 26349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:35.315886 26349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:35.315975 26349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:35.325732 26349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:35.327394 26349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:35.328758 26349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:35.340265 26357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:35.340348 26354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:35.340335 26355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:35.344808 26349 server_base.cc:1034] running on GCE node
I20260926 07:33:35.345646 26349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:35.347023 26349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:35.348279 26349 hybrid_clock.cc:648] HybridClock initialized: now 1790408015348234 us; error 59 us; skew 500 ppm
I20260926 07:33:35.348626 26349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:35 dist-test-slave-qlq8 krb5kdc[26336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408015, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:35.358305 26349 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:35.359997 26349 webserver.cc:460] Webserver started at http://127.22.221.126:38573/ using document root <none> and password file <none>
I20260926 07:33:35.360710 26349 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:35.360855 26349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:35.361167 26349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:35.364192 26349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "eae2ff0cfb8945e3a930a0dc17022074"
format_stamp: "Formatted at 2026-09-26 07:33:35 on dist-test-slave-qlq8"
server_key: "c36a52e4241cea52224da03a83d81e58"
server_key_iv: "8b50d6010277560c64e92aa0386aed78"
server_key_version: "encryptionkey@0"
I20260926 07:33:35.364974 26349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "eae2ff0cfb8945e3a930a0dc17022074"
format_stamp: "Formatted at 2026-09-26 07:33:35 on dist-test-slave-qlq8"
server_key: "c36a52e4241cea52224da03a83d81e58"
server_key_iv: "8b50d6010277560c64e92aa0386aed78"
server_key_version: "encryptionkey@0"
I20260926 07:33:35.369953 26349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:35.373245 26364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:35.374531 26349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:35.374703 26349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "eae2ff0cfb8945e3a930a0dc17022074"
format_stamp: "Formatted at 2026-09-26 07:33:35 on dist-test-slave-qlq8"
server_key: "c36a52e4241cea52224da03a83d81e58"
server_key_iv: "8b50d6010277560c64e92aa0386aed78"
server_key_version: "encryptionkey@0"
I20260926 07:33:35.374897 26349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:35.403834 26349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:35.408417 26349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:35.408720 26349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:35.426851 26349 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:38171
I20260926 07:33:35.426875 26415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:38171 every 8 connection(s)
I20260926 07:33:35.428237 26349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:35.431919 26416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:35.436360 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26349
I20260926 07:33:35.436595 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:35.437244 23413 external_mini_cluster.cc:1428] Setting key e94078ce0e36c07808678a10a9f23472
I20260926 07:33:35.442598 26416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074: Bootstrap starting.
I20260926 07:33:35.446269 26416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:35.447525 26416 log.cc:824] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:35.450863 26416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074: No bootstrap required, opened a new log
Sep 26 07:33:35 dist-test-slave-qlq8 krb5kdc[26336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:35.459046 26416 raft_consensus.cc:348] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae2ff0cfb8945e3a930a0dc17022074" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 38171 } }
I20260926 07:33:35.459517 26416 raft_consensus.cc:374] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:35.459663 26416 raft_consensus.cc:729] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae2ff0cfb8945e3a930a0dc17022074, State: Initialized, Role: FOLLOWER
I20260926 07:33:35.460229 26416 consensus_queue.cc:260] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae2ff0cfb8945e3a930a0dc17022074" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 38171 } }
I20260926 07:33:35.460498 26416 raft_consensus.cc:388] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:35.460650 26416 raft_consensus.cc:482] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:35.460848 26416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:35.461207 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:35.439126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.22.221.126:38171)
0926 07:33:35.440115 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:35.440126 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:35.440171 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:33:35.441873 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:35.441894 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:35.442274 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:35.442595 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:35.444479 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:35.445813 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:35.447655 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:35.448714 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:35.448915 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:35.453544 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:35.453555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:35.453569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:35.453612 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:35.456773 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:35.457492 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:35.457498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:35.457504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:35.457674 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:35.458095 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:35.458098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:35.458101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:35.458632 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:35.458923 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:35.459439 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 07:33:35.459913 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":430,"threads_started":1}
I20260926 07:33:35.463435 26416 raft_consensus.cc:504] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae2ff0cfb8945e3a930a0dc17022074" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 38171 } }
I20260926 07:33:35.464021 26416 leader_election.cc:302] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eae2ff0cfb8945e3a930a0dc17022074; no voters:
I20260926 07:33:35.464727 26416 leader_election.cc:288] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:35.464907 26421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:35.465322 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:33:35.466936 26416 sys_catalog.cc:564] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:35.470891 26421 raft_consensus.cc:686] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [term 1 LEADER]: Becoming Leader. State: Replica: eae2ff0cfb8945e3a930a0dc17022074, State: Running, Role: LEADER
I20260926 07:33:35.471634 26421 consensus_queue.cc:237] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae2ff0cfb8945e3a930a0dc17022074" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 38171 } }
I20260926 07:33:35.474473 26422 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eae2ff0cfb8945e3a930a0dc17022074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae2ff0cfb8945e3a930a0dc17022074" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 38171 } } }
I20260926 07:33:35.474918 26422 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:35.475425 26423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eae2ff0cfb8945e3a930a0dc17022074. Latest consensus state: current_term: 1 leader_uuid: "eae2ff0cfb8945e3a930a0dc17022074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae2ff0cfb8945e3a930a0dc17022074" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 38171 } } }
I20260926 07:33:35.475740 26423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:35.480016 26433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:35.493628 26433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:35.505663 26433 catalog_manager.cc:1347] Generated new cluster ID: 07a73a9273c14fb787ed17de74f271e8
I20260926 07:33:35.505808 26433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:35.518788 26433 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:35.520402 26433 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:35.529215 26433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eae2ff0cfb8945e3a930a0dc17022074: Generated new TSK 0
I20260926 07:33:35.529954 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:35.520309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.22.221.126:38171)
0926 07:33:35.520512 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:35.520519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:35.521088 (+ 569us) server_negotiation.cc:407] Connection header received
0926 07:33:35.521293 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:35.521300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:35.521411 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:35.521548 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:35.522834 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:35.524070 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:35.525960 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:35.526624 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:35.526769 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:35.528089 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:35.528096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:35.528101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:35.528961 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:35.529038 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:35.529074 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:35.529168 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:35.529445 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:33:35.529563 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":73}
I20260926 07:33:35.530123 26433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:33:35.543787 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26349
I20260926 07:33:35.544930 26411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:35.633284 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26349
2026-09-26T07:33:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1931 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:35 dist-test-slave-qlq8 krb5kdc[26451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:35 dist-test-slave-qlq8 krb5kdc[26451](info): set up 2 sockets
Sep 26 07:33:35 dist-test-slave-qlq8 krb5kdc[26451](info): commencing operation
krb5kdc: starting...
W20260926 07:33:36.723737 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:36 dist-test-slave-qlq8 krb5kdc[26451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:36.927860 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43565
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:37.244891 26464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:37.245244 26464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:37.245415 26464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:37.252594 26464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:37.252727 26464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:37.252835 26464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:37.252930 26464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:37.253034 26464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:37.262861 26464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:37.264218 26464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:37.266083 26464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:37.278425 26469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:37.278383 26472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:37.278465 26470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:37.279556 26464 server_base.cc:1034] running on GCE node
I20260926 07:33:37.280340 26464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:37.281720 26464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:37.282961 26464 hybrid_clock.cc:648] HybridClock initialized: now 1790408017282908 us; error 60 us; skew 500 ppm
I20260926 07:33:37.283381 26464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:37 dist-test-slave-qlq8 krb5kdc[26451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408017, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:37.286386 26464 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:37.287978 26464 webserver.cc:460] Webserver started at http://127.22.221.126:44429/ using document root <none> and password file <none>
I20260926 07:33:37.288666 26464 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:37.288820 26464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:37.289126 26464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:37.292098 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "252c052a46bd4079a2810ef6cb4b4010"
format_stamp: "Formatted at 2026-09-26 07:33:37 on dist-test-slave-qlq8"
server_key: "fd486683344a69c8cc069c484a0aa1ef"
server_key_iv: "1a70829dbd00e6d7aa78b993fd5a12e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:37.292861 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "252c052a46bd4079a2810ef6cb4b4010"
format_stamp: "Formatted at 2026-09-26 07:33:37 on dist-test-slave-qlq8"
server_key: "fd486683344a69c8cc069c484a0aa1ef"
server_key_iv: "1a70829dbd00e6d7aa78b993fd5a12e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:37.297868 26464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:37.301199 26479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:37.302502 26464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:37.302687 26464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "252c052a46bd4079a2810ef6cb4b4010"
format_stamp: "Formatted at 2026-09-26 07:33:37 on dist-test-slave-qlq8"
server_key: "fd486683344a69c8cc069c484a0aa1ef"
server_key_iv: "1a70829dbd00e6d7aa78b993fd5a12e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:37.302881 26464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:37.320928 26464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:37.324780 26464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:37.325070 26464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:37.343266 26464 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:43565
I20260926 07:33:37.343283 26530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:43565 every 8 connection(s)
I20260926 07:33:37.344633 26464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:37.345131 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26464
I20260926 07:33:37.345351 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:37.346127 23413 external_mini_cluster.cc:1428] Setting key d7624ca91e6043e2e62cb66260208bc5
I20260926 07:33:37.348867 26531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:33:37 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:37.361775 26531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010: Bootstrap starting.
I20260926 07:33:37.365152 26531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:37.366191 26531 log.cc:824] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:37.368758 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:37.348100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.22.221.126:43565)
0926 07:33:37.348897 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:37.348908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:37.348951 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:33:37.350549 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:37.350568 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:37.350923 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:37.351315 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:37.353022 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:37.354348 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:37.356113 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:37.356748 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:37.356873 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:37.361400 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:37.361411 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:37.361423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:37.361456 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:37.364467 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:37.365125 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:37.365131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:37.365137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:37.365334 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:37.365727 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:37.365732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:37.365735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:37.366244 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:37.366555 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:37.367007 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 07:33:37.367467 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":350,"threads_started":1}
I20260926 07:33:37.369493 26531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010: No bootstrap required, opened a new log
I20260926 07:33:37.378086 26531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c052a46bd4079a2810ef6cb4b4010" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43565 } }
I20260926 07:33:37.378456 26531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:37.378585 26531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252c052a46bd4079a2810ef6cb4b4010, State: Initialized, Role: FOLLOWER
I20260926 07:33:37.379140 26531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c052a46bd4079a2810ef6cb4b4010" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43565 } }
I20260926 07:33:37.379452 26531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:37.379601 26531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:37.379798 26531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:37.381896 26531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c052a46bd4079a2810ef6cb4b4010" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43565 } }
I20260926 07:33:37.382397 26531 leader_election.cc:302] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252c052a46bd4079a2810ef6cb4b4010; no voters:
I20260926 07:33:37.382962 26531 leader_election.cc:288] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:37.383093 26536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:37.384971 26531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:37.385326 26536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [term 1 LEADER]: Becoming Leader. State: Replica: 252c052a46bd4079a2810ef6cb4b4010, State: Running, Role: LEADER
I20260926 07:33:37.385835 26536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c052a46bd4079a2810ef6cb4b4010" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43565 } }
I20260926 07:33:37.388939 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252c052a46bd4079a2810ef6cb4b4010. Latest consensus state: current_term: 1 leader_uuid: "252c052a46bd4079a2810ef6cb4b4010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c052a46bd4079a2810ef6cb4b4010" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43565 } } }
I20260926 07:33:37.389284 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:37.389038 26537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252c052a46bd4079a2810ef6cb4b4010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c052a46bd4079a2810ef6cb4b4010" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43565 } } }
I20260926 07:33:37.389467 26537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:37.393256 26546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:37.397508 26546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:37.406050 26546 catalog_manager.cc:1347] Generated new cluster ID: d7d8c0940bb042409f15577c0f1e216e
I20260926 07:33:37.406172 26546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:37.424996 26546 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:37.426955 26546 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:37.435686 26546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252c052a46bd4079a2810ef6cb4b4010: Generated new TSK 0
I20260926 07:33:37.436754 26546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:37.486599 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43565
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:37.804899 26557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:37.805218 26557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:37.805388 26557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:37.812817 26557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:37.812966 26557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:37.813185 26557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:37.822697 26557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:37.824060 26557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:37.825418 26557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:37.838697 26563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:37.838795 26565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:37.838759 26562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:37.839603 26557 server_base.cc:1034] running on GCE node
I20260926 07:33:37.840193 26557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:37.841097 26557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:37.842319 26557 hybrid_clock.cc:648] HybridClock initialized: now 1790408017842283 us; error 60 us; skew 500 ppm
I20260926 07:33:37.842629 26557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:37 dist-test-slave-qlq8 krb5kdc[26451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408017, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:37.846060 26557 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:37.847515 26557 webserver.cc:460] Webserver started at http://127.22.221.65:33567/ using document root <none> and password file <none>
I20260926 07:33:37.848191 26557 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:37.848299 26557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:37.848594 26557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:37.851461 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "92887ec9352b4387b56926340fc12950"
format_stamp: "Formatted at 2026-09-26 07:33:37 on dist-test-slave-qlq8"
server_key: "26a80dfde2955da2282ba442ac8858db"
server_key_iv: "5b1fd22aa217124be56368d4494c452c"
server_key_version: "encryptionkey@0"
I20260926 07:33:37.852171 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "92887ec9352b4387b56926340fc12950"
format_stamp: "Formatted at 2026-09-26 07:33:37 on dist-test-slave-qlq8"
server_key: "26a80dfde2955da2282ba442ac8858db"
server_key_iv: "5b1fd22aa217124be56368d4494c452c"
server_key_version: "encryptionkey@0"
I20260926 07:33:37.857107 26557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:33:37.860283 26572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:37.861524 26557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:33:37.861646 26557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "92887ec9352b4387b56926340fc12950"
format_stamp: "Formatted at 2026-09-26 07:33:37 on dist-test-slave-qlq8"
server_key: "26a80dfde2955da2282ba442ac8858db"
server_key_iv: "5b1fd22aa217124be56368d4494c452c"
server_key_version: "encryptionkey@0"
I20260926 07:33:37.861795 26557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:37.872746 26557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:37.876675 26557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:37.876916 26557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:37.877786 26557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:37.878991 26557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:37.879061 26557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:37.879140 26557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:37.879182 26557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:37.912613 26557 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:33427
I20260926 07:33:37.912654 26684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:33427 every 8 connection(s)
I20260926 07:33:37.913961 26557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:37.916158 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26557
I20260926 07:33:37.916375 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:37.917210 23413 external_mini_cluster.cc:1428] Setting key 0c8227d7c8bf778802018e6886a272f1
Sep 26 07:33:37 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408017, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:37.934607 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:37.916630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:38631 (local address 127.22.221.126:43565)
0926 07:33:37.917094 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:37.917098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:37.918314 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 07:33:37.920239 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:37.920245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:37.920335 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:37.920453 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:37.921946 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:37.922657 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:37.924774 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:37.925173 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:37.925279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:37.928726 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:37.928731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:37.928736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:37.928767 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:37.931384 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:37.932239 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:37.932244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:37.932248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:37.932377 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:37.932829 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:37.932834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:37.932837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:37.933139 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:37.933307 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:37.933997 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 07:33:37.934239 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":260,"threads_started":1}
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:37.935808 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:37.917319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43565 (local address 127.22.221.65:38631)
0926 07:33:37.918154 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:37.918198 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:37.919967 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:37.920689 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:37.920697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:37.921210 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:37.921771 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:37.921788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:37.922821 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:37.922831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:37.924589 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:37.924630 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:37.925369 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:37.925373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:37.925607 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:37.926290 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:37.926313 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:37.928488 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:37.931524 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:37.931533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:37.931550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:37.932089 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:37.932499 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:37.932504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:37.932507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:37.932699 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:37.933276 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:37.933284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:37.933628 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 07:33:37.933901 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 07:33:37.934448 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":307,"threads_started":1}
I20260926 07:33:37.937398 26685 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43565
I20260926 07:33:37.937717 26685 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:33:37.938612 26685 heartbeater.cc:511] Master 127.22.221.126:43565 requested a full tablet report, sending...
I20260926 07:33:37.940804 26496 ts_manager.cc:194] Registered new tserver with Master: 92887ec9352b4387b56926340fc12950 (127.22.221.65:33427)
I20260926 07:33:37.942214 26496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:38631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:37.962086 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43565
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:38.281317 26691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:38.281592 26691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:38.281793 26691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:38.289523 26691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:38.289655 26691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:38.289839 26691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:38.299643 26691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:38.300909 26691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:38.302232 26691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:38.315585 26699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:38.315670 26696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:38.315646 26697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:38.316398 26691 server_base.cc:1034] running on GCE node
I20260926 07:33:38.316928 26691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:38.317826 26691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:38.319068 26691 hybrid_clock.cc:648] HybridClock initialized: now 1790408018319014 us; error 47 us; skew 500 ppm
I20260926 07:33:38.319434 26691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:38.322577 26691 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:38.324139 26691 webserver.cc:460] Webserver started at http://127.22.221.66:36225/ using document root <none> and password file <none>
I20260926 07:33:38.324838 26691 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:38.324975 26691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:38.325299 26691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:38.328296 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "ae7e61a1395f401c9c58c37f42e2880f"
format_stamp: "Formatted at 2026-09-26 07:33:38 on dist-test-slave-qlq8"
server_key: "8ebf7220cabc685bfee7e17710c35c05"
server_key_iv: "194da82b426182b5c398b0620f1d72d8"
server_key_version: "encryptionkey@0"
I20260926 07:33:38.329094 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "ae7e61a1395f401c9c58c37f42e2880f"
format_stamp: "Formatted at 2026-09-26 07:33:38 on dist-test-slave-qlq8"
server_key: "8ebf7220cabc685bfee7e17710c35c05"
server_key_iv: "194da82b426182b5c398b0620f1d72d8"
server_key_version: "encryptionkey@0"
I20260926 07:33:38.334223 26691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:38.337689 26706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:38.339111 26691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:38.339362 26691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "ae7e61a1395f401c9c58c37f42e2880f"
format_stamp: "Formatted at 2026-09-26 07:33:38 on dist-test-slave-qlq8"
server_key: "8ebf7220cabc685bfee7e17710c35c05"
server_key_iv: "194da82b426182b5c398b0620f1d72d8"
server_key_version: "encryptionkey@0"
I20260926 07:33:38.339583 26691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:38.355177 26691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:38.360080 26691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:38.360375 26691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:38.361438 26691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:38.362867 26691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:38.362968 26691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:38.363118 26691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:38.363191 26691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:38.397019 26691 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:33029
I20260926 07:33:38.397064 26818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:33029 every 8 connection(s)
I20260926 07:33:38.398357 26691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:38.400866 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26691
I20260926 07:33:38.401067 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:38.401530 23413 external_mini_cluster.cc:1428] Setting key a495580ae0964271d4cdcb5d3ae9762f
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:33:38.418957 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:38.400982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:39687 (local address 127.22.221.126:43565)
0926 07:33:38.402199 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:38.402207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:38.402553 (+ 346us) server_negotiation.cc:407] Connection header received
0926 07:33:38.404459 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:38.404464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:38.404557 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:38.404666 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:38.406165 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.407014 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.408792 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.409226 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.409364 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:38.413127 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:38.413132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:38.413135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:38.413172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:38.415933 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.416670 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.416680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.416683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.416818 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.417257 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.417266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.417268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.417516 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:38.417683 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:38.418371 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 07:33:38.418608 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068}
I20260926 07:33:38.420001 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:38.401530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43565 (local address 127.22.221.66:39687)
0926 07:33:38.402385 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:38.402436 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:38.404194 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:38.404907 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:38.404915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:38.405408 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:38.405986 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:38.406005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.407176 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.407180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:38.408639 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:38.408652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.409453 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.409460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:38.409706 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:38.410427 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:38.410450 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:38.412892 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:38.416079 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:38.416088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:38.416105 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:38.416540 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:38.416931 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:38.416936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:38.416940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:38.417130 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:38.417658 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:38.417666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:38.418009 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 07:33:38.418268 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 07:33:38.418796 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":368,"threads_started":1}
I20260926 07:33:38.421573 26819 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
I20260926 07:33:38.421977 26819 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:38.422766 26819 heartbeater.cc:511] Master 127.22.221.126:43565 requested a full tablet report, sending...
I20260926 07:33:38.424423 26496 ts_manager.cc:194] Registered new tserver with Master: ae7e61a1395f401c9c58c37f42e2880f (127.22.221.66:33029)
I20260926 07:33:38.425715 26496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:39687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:38.446173 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43565
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:38.770905 26824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:38.771181 26824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:38.771369 26824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:38.778687 26824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:38.778822 26824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:38.778964 26824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:38.788846 26824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:38.790143 26824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:38.791428 26824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:38.804657 26829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:38.804605 26832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:38.804636 26830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:38.805408 26824 server_base.cc:1034] running on GCE node
I20260926 07:33:38.806032 26824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:38.807030 26824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:38.808230 26824 hybrid_clock.cc:648] HybridClock initialized: now 1790408018808148 us; error 68 us; skew 500 ppm
I20260926 07:33:38.808573 26824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:38.811615 26824 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:38.813074 26824 webserver.cc:460] Webserver started at http://127.22.221.67:36523/ using document root <none> and password file <none>
I20260926 07:33:38.813760 26824 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:38.813905 26824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:38.814213 26824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:38.817087 26824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "dd295cbcb4e34b79a6bb9eb744c894bf"
format_stamp: "Formatted at 2026-09-26 07:33:38 on dist-test-slave-qlq8"
server_key: "bc7743be40cbac728eeed48bc1c69974"
server_key_iv: "93cac8e77bef476ab1d7d50286175ee2"
server_key_version: "encryptionkey@0"
I20260926 07:33:38.817876 26824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "dd295cbcb4e34b79a6bb9eb744c894bf"
format_stamp: "Formatted at 2026-09-26 07:33:38 on dist-test-slave-qlq8"
server_key: "bc7743be40cbac728eeed48bc1c69974"
server_key_iv: "93cac8e77bef476ab1d7d50286175ee2"
server_key_version: "encryptionkey@0"
I20260926 07:33:38.822988 26824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:33:38.826516 26839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:38.827946 26824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:38.828122 26824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "dd295cbcb4e34b79a6bb9eb744c894bf"
format_stamp: "Formatted at 2026-09-26 07:33:38 on dist-test-slave-qlq8"
server_key: "bc7743be40cbac728eeed48bc1c69974"
server_key_iv: "93cac8e77bef476ab1d7d50286175ee2"
server_key_version: "encryptionkey@0"
I20260926 07:33:38.828336 26824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:38.848793 26824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:38.852720 26824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:38.853021 26824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:38.854004 26824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:38.855332 26824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:38.855430 26824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:38.855583 26824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:38.855656 26824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:38.888530 26824 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:43497
I20260926 07:33:38.888585 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:43497 every 8 connection(s)
I20260926 07:33:38.889885 26824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:38.894358 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26824
I20260926 07:33:38.894557 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:38.895165 23413 external_mini_cluster.cc:1428] Setting key 965d69946ae18658a4c4fea1ebecb35e
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:38.910928 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:38.892537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:41313 (local address 127.22.221.126:43565)
0926 07:33:38.892722 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:38.892728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:38.894024 (+ 1296us) server_negotiation.cc:407] Connection header received
0926 07:33:38.896573 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:38.896579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:38.896666 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:38.896777 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:38.898562 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.899355 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.901024 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.901452 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.901588 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:38.905049 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:38.905054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:38.905058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:38.905089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:38.907763 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.908577 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.908580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.908583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.908701 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.909194 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.909198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.909200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.909487 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:38.909645 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:38.910339 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 07:33:38.910551 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:33:38.912016 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:38.893085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43565 (local address 127.22.221.67:41313)
0926 07:33:38.893798 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:38.893841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:38.896238 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:38.897034 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:38.897042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:38.897698 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:38.898393 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:38.898417 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.899506 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.899510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:38.900871 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:38.900883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.901691 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.901699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:38.901950 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:38.902582 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:38.902605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:38.904844 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:38.907888 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:38.907895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:38.907908 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:38.908464 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:38.908817 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:38.908821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:38.908824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:38.909059 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:38.909621 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:38.909629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:38.909988 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 07:33:38.910259 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 07:33:38.910807 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":279,"threads_started":1}
I20260926 07:33:38.913519 26952 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43565
I20260926 07:33:38.913831 26952 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:38.914541 26952 heartbeater.cc:511] Master 127.22.221.126:43565 requested a full tablet report, sending...
I20260926 07:33:38.916026 26496 ts_manager.cc:194] Registered new tserver with Master: dd295cbcb4e34b79a6bb9eb744c894bf (127.22.221.67:43497)
I20260926 07:33:38.917132 26496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:41313
I20260926 07:33:38.923156 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:38.945221 26685 heartbeater.cc:503] Master 127.22.221.126:43565 was elected leader, sending a full tablet report...
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:38.970904 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:38.957339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.22.221.126:43565)
0926 07:33:38.957520 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:38.957524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:38.958004 (+ 480us) server_negotiation.cc:407] Connection header received
0926 07:33:38.958161 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:38.958164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:38.958232 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:38.958323 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:38.959808 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.960550 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.961785 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.962176 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.962275 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:38.966213 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:38.966218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:38.966222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:38.966273 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:38.968731 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.969312 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.969316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.969318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.969454 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.969814 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.969817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.969819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.970114 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:38.970286 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:38.970519 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:33:38.970675 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:33:38.971546 26496 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49366
I20260926 07:33:38.972996 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26557
Sep 26 07:33:38 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:38.988632 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:38.974242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.22.221.65:33427)
0926 07:33:38.974853 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:38.974859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:38.974883 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:33:38.974938 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:38.974945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:38.975145 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:38.975346 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:38.976415 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.977405 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.978537 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:38.979081 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:38.979173 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:38.982887 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:38.982899 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:38.982912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:38.982947 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:38.985719 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.986397 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.986404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.986406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.986527 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:38.986905 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:38.986911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:38.986914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:38.987302 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:38.987603 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:38.987913 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 07:33:38.988202 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":313,"threads_started":1}
W20260926 07:33:38.989851 26680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58204
W20260926 07:33:38.990953 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26557: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:38.991153 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26557
I20260926 07:33:39.006784 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26691
Sep 26 07:33:39 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:39.022451 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:39.007727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.22.221.66:33029)
0926 07:33:39.008324 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:39.008331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:39.008358 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:33:39.008438 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:39.008445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:39.008758 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:39.008892 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:39.010130 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:39.011108 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:39.012366 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:39.012910 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:39.013019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:39.017056 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:39.017067 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:39.017080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:39.017113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:39.020061 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:39.020656 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:39.020663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:39.020665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:39.020784 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:39.021091 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:39.021094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:39.021096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:39.021415 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:39.021634 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:39.021848 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:33:39.022036 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":291,"threads_started":1}
W20260926 07:33:39.023741 26814 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51692
W20260926 07:33:39.024847 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:39.024998 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26691
I20260926 07:33:39.041240 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26824
Sep 26 07:33:39 dist-test-slave-qlq8 krb5kdc[26451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408018, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:39.057844 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:39.042237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.22.221.67:43497)
0926 07:33:39.042916 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:39.042923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:39.042953 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:33:39.043014 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:39.043020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:39.043308 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:39.043445 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:39.044752 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:39.045837 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:39.047006 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:39.047651 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:39.047756 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:39.051895 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:39.051907 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:39.051923 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:39.051964 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:39.055335 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:39.056042 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:39.056049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:39.056052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:39.056194 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:39.056523 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:39.056526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:39.056528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:39.056877 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:39.057109 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:39.057327 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:33:39.057530 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":358,"threads_started":1}
W20260926 07:33:39.059154 26947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38098
W20260926 07:33:39.060356 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26824: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:33:39.060508 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26824
I20260926 07:33:39.075702 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26464
I20260926 07:33:39.077013 26526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:39.185241 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26464
2026-09-26T07:33:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3557 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:39 dist-test-slave-qlq8 krb5kdc[26969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:39 dist-test-slave-qlq8 krb5kdc[26969](info): set up 2 sockets
Sep 26 07:33:39 dist-test-slave-qlq8 krb5kdc[26969](info): commencing operation
krb5kdc: starting...
W20260926 07:33:40.280500 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:40 dist-test-slave-qlq8 krb5kdc[26969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:40.483574 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:41743
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:41743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:40.805660 26982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:40.805994 26982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:40.806160 26982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:40.813722 26982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:40.813865 26982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:40.813969 26982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:40.814062 26982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:40.814146 26982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:40.824374 26982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:41743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:40.825744 26982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:40.827121 26982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:40.838828 26987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:40.838748 26988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:40.838789 26990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:40.839596 26982 server_base.cc:1034] running on GCE node
I20260926 07:33:40.840418 26982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:40.842056 26982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:40.843322 26982 hybrid_clock.cc:648] HybridClock initialized: now 1790408020843284 us; error 61 us; skew 500 ppm
I20260926 07:33:40.843672 26982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:40 dist-test-slave-qlq8 krb5kdc[26969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:40.852718 26982 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:40.854234 26982 webserver.cc:460] Webserver started at http://127.22.221.126:34199/ using document root <none> and password file <none>
I20260926 07:33:40.854950 26982 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:40.855106 26982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:40.855513 26982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:40.858448 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7"
format_stamp: "Formatted at 2026-09-26 07:33:40 on dist-test-slave-qlq8"
server_key: "e8683476ad7c9762b741f5cb4791a2de"
server_key_iv: "eb4009f80a1bc229d52ed30594750b7d"
server_key_version: "encryptionkey@0"
I20260926 07:33:40.859274 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7"
format_stamp: "Formatted at 2026-09-26 07:33:40 on dist-test-slave-qlq8"
server_key: "e8683476ad7c9762b741f5cb4791a2de"
server_key_iv: "eb4009f80a1bc229d52ed30594750b7d"
server_key_version: "encryptionkey@0"
I20260926 07:33:40.864310 26982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:33:40.867700 26997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:40.868943 26982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:33:40.869134 26982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7"
format_stamp: "Formatted at 2026-09-26 07:33:40 on dist-test-slave-qlq8"
server_key: "e8683476ad7c9762b741f5cb4791a2de"
server_key_iv: "eb4009f80a1bc229d52ed30594750b7d"
server_key_version: "encryptionkey@0"
I20260926 07:33:40.869335 26982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:40.888259 26982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:40.892781 26982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:40.893095 26982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:40.911371 26982 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:41743
I20260926 07:33:40.911415 27048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:41743 every 8 connection(s)
I20260926 07:33:40.912762 26982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:40.916587 27049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:40.921832 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 26982
I20260926 07:33:40.922061 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:40.922683 23413 external_mini_cluster.cc:1428] Setting key c2421e5c8756bd489d6bdfe16dbb88f4
I20260926 07:33:40.927405 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7: Bootstrap starting.
I20260926 07:33:40.931030 27049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:40.932271 27049 log.cc:824] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:40.935158 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7: No bootstrap required, opened a new log
Sep 26 07:33:40 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:40.943683 27049 raft_consensus.cc:348] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 41743 } }
I20260926 07:33:40.944093 27049 raft_consensus.cc:374] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:40.944233 27049 raft_consensus.cc:729] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acbcdc180d2d471a9c1a70ebc60a1bf7, State: Initialized, Role: FOLLOWER
I20260926 07:33:40.944927 27049 consensus_queue.cc:260] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 41743 } }
I20260926 07:33:40.945199 27049 raft_consensus.cc:388] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:40.945358 27049 raft_consensus.cc:482] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:40.945555 27049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:40.945881 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:40.924777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.22.221.126:41743)
0926 07:33:40.925656 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:40.925666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:40.925707 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:33:40.927458 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:40.927506 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:40.927913 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:40.928254 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:40.930063 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:40.931490 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:40.933319 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:40.933973 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:40.934171 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:40.938335 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:40.938342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:40.938358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:40.938401 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:40.941563 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:40.942268 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:40.942281 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:40.942287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:40.942454 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:40.942862 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:40.942874 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:40.942877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:40.943502 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:40.943801 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:40.944257 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 07:33:40.944720 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":425,"threads_started":1}
I20260926 07:33:40.948518 27049 raft_consensus.cc:504] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 41743 } }
I20260926 07:33:40.949183 27049 leader_election.cc:302] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acbcdc180d2d471a9c1a70ebc60a1bf7; no voters:
I20260926 07:33:40.949688 27049 leader_election.cc:288] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:40.949837 27054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:40.951490 27054 raft_consensus.cc:686] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [term 1 LEADER]: Becoming Leader. State: Replica: acbcdc180d2d471a9c1a70ebc60a1bf7, State: Running, Role: LEADER
I20260926 07:33:40.951960 27049 sys_catalog.cc:564] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:40.952097 27054 consensus_queue.cc:237] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 41743 } }
I20260926 07:33:40.955050 27056 sys_catalog.cc:455] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acbcdc180d2d471a9c1a70ebc60a1bf7. Latest consensus state: current_term: 1 leader_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 41743 } } }
I20260926 07:33:40.955715 27056 sys_catalog.cc:458] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:40.955188 27055 sys_catalog.cc:455] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbcdc180d2d471a9c1a70ebc60a1bf7" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 41743 } } }
I20260926 07:33:40.956265 27055 sys_catalog.cc:458] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:40.959898 27063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:40.966171 27063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:40.974823 27063 catalog_manager.cc:1347] Generated new cluster ID: aff75cf28c2c4b01babdba66ffc856c3
I20260926 07:33:40.974936 27063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:40.998948 27063 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:41.000998 27063 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:41.009425 27063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acbcdc180d2d471a9c1a70ebc60a1bf7: Generated new TSK 0
I20260926 07:33:41.010454 27063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:41.059510 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:41743
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:41.382730 27075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:41.383046 27075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:41.383266 27075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:41.390635 27075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:41.390792 27075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:41.390969 27075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:41.400858 27075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:41743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:41.402135 27075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:41.403474 27075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:41.416738 27081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:41.416837 27080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:41.416715 27083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:41.417326 27075 server_base.cc:1034] running on GCE node
I20260926 07:33:41.418078 27075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:41.419009 27075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:41.420243 27075 hybrid_clock.cc:648] HybridClock initialized: now 1790408021420184 us; error 50 us; skew 500 ppm
I20260926 07:33:41.420578 27075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:41 dist-test-slave-qlq8 krb5kdc[26969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408021, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:41.423846 27075 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:41.425339 27075 webserver.cc:460] Webserver started at http://127.22.221.65:44709/ using document root <none> and password file <none>
I20260926 07:33:41.426088 27075 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:41.426234 27075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:41.426537 27075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:41.429525 27075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "eebd43f2c23e480aa82ee0a3e8f503c0"
format_stamp: "Formatted at 2026-09-26 07:33:41 on dist-test-slave-qlq8"
server_key: "90719a2a22b2aebd78dfc1689af976d4"
server_key_iv: "d3af7f47d2a54a3afc7b8a703faa3dee"
server_key_version: "encryptionkey@0"
I20260926 07:33:41.430321 27075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "eebd43f2c23e480aa82ee0a3e8f503c0"
format_stamp: "Formatted at 2026-09-26 07:33:41 on dist-test-slave-qlq8"
server_key: "90719a2a22b2aebd78dfc1689af976d4"
server_key_iv: "d3af7f47d2a54a3afc7b8a703faa3dee"
server_key_version: "encryptionkey@0"
I20260926 07:33:41.435607 27075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:33:41.439082 27090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:41.440433 27075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:41.440611 27075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "eebd43f2c23e480aa82ee0a3e8f503c0"
format_stamp: "Formatted at 2026-09-26 07:33:41 on dist-test-slave-qlq8"
server_key: "90719a2a22b2aebd78dfc1689af976d4"
server_key_iv: "d3af7f47d2a54a3afc7b8a703faa3dee"
server_key_version: "encryptionkey@0"
I20260926 07:33:41.440825 27075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:41.471144 27075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:41.475301 27075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:41.475587 27075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:41.476545 27075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:41.477857 27075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:41.477962 27075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:41.478107 27075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:41.478188 27075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:41.511168 27075 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:38133
I20260926 07:33:41.511286 27202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:38133 every 8 connection(s)
I20260926 07:33:41.512478 27075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:41.520977 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27075
I20260926 07:33:41.521204 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:41.521821 23413 external_mini_cluster.cc:1428] Setting key ba5bb0000898849752f5eb42b0d35cfe
Sep 26 07:33:41 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408021, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:41.533850 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:41.515089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:47187 (local address 127.22.221.126:41743)
0926 07:33:41.515542 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:41.515546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:41.516533 (+ 987us) server_negotiation.cc:407] Connection header received
0926 07:33:41.518325 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:41.518331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:41.518421 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:41.518541 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:41.520078 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:41.520818 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:41.522600 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:41.523122 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:41.523284 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:41.527367 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:41.527375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:41.527382 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:41.527425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:41.530455 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:41.531196 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:41.531212 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:41.531219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:41.531401 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:41.531830 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:41.531838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:41.531844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:41.532223 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:41.532375 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:41.533080 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 07:33:41.533348 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260926 07:33:41.534503 27206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:41.515705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:41743 (local address 127.22.221.65:47187)
0926 07:33:41.516378 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:41.516416 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:41.518061 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:41.518781 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:41.518788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:41.519352 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:41.519907 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:41.519924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:41.520975 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:41.520979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:41.522419 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:41.522437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:41.523379 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:41.523387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:41.523709 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:41.524448 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:41.524469 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:41.527119 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:41.530614 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:41.530621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:41.530633 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:41.531060 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:41.531520 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:41.531524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:41.531526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:41.531710 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:41.532341 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:41.532347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:41.532652 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 07:33:41.532885 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 07:33:41.533301 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":256,"threads_started":1}
I20260926 07:33:41.536083 27203 heartbeater.cc:347] Connected to a master server at 127.22.221.126:41743
I20260926 07:33:41.536405 27203 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:41.537143 27203 heartbeater.cc:511] Master 127.22.221.126:41743 requested a full tablet report, sending...
I20260926 07:33:41.539177 27014 ts_manager.cc:194] Registered new tserver with Master: eebd43f2c23e480aa82ee0a3e8f503c0 (127.22.221.65:38133)
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:41.540642 27014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:47187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:41.566028 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:41743
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:41.876246 27209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:41.876560 27209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:41.876747 27209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:41.883806 27209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:41.883961 27209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:41.884147 27209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:41.893735 27209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:41743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:41.895067 27209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:41.896422 27209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:41.909264 27215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:41.909340 27214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:41.909314 27217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:41.910166 27209 server_base.cc:1034] running on GCE node
I20260926 07:33:41.910899 27209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:41.912087 27209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:41.913331 27209 hybrid_clock.cc:648] HybridClock initialized: now 1790408021913269 us; error 53 us; skew 500 ppm
I20260926 07:33:41.913673 27209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:41 dist-test-slave-qlq8 krb5kdc[26969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408021, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:41.917311 27209 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:41.924572 27209 webserver.cc:460] Webserver started at http://127.22.221.66:33411/ using document root <none> and password file <none>
I20260926 07:33:41.925309 27209 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:41.925463 27209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:41.925778 27209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:41.928753 27209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "2ef274a0583f4db689f85292e8915028"
format_stamp: "Formatted at 2026-09-26 07:33:41 on dist-test-slave-qlq8"
server_key: "2fc73cd3ab5b2fb2d862e91a8ec63008"
server_key_iv: "4ff2aa268af5e91d1230950877797ea7"
server_key_version: "encryptionkey@0"
I20260926 07:33:41.929530 27209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "2ef274a0583f4db689f85292e8915028"
format_stamp: "Formatted at 2026-09-26 07:33:41 on dist-test-slave-qlq8"
server_key: "2fc73cd3ab5b2fb2d862e91a8ec63008"
server_key_iv: "4ff2aa268af5e91d1230950877797ea7"
server_key_version: "encryptionkey@0"
I20260926 07:33:41.934768 27209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:33:41.938220 27224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:41.939725 27209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:41.939911 27209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "2ef274a0583f4db689f85292e8915028"
format_stamp: "Formatted at 2026-09-26 07:33:41 on dist-test-slave-qlq8"
server_key: "2fc73cd3ab5b2fb2d862e91a8ec63008"
server_key_iv: "4ff2aa268af5e91d1230950877797ea7"
server_key_version: "encryptionkey@0"
I20260926 07:33:41.940119 27209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:41.952877 27209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:41.957391 27209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:41.957686 27209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:41.958611 27209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:41.960000 27209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:41.960104 27209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:41.960259 27209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:41.960333 27209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:41.992928 27209 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:34653
I20260926 07:33:41.992983 27336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:34653 every 8 connection(s)
I20260926 07:33:41.994236 27209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:42.003731 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27209
I20260926 07:33:42.003922 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:42.004354 23413 external_mini_cluster.cc:1428] Setting key 05ed16f981710598f248c330a4ec1a22
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408021, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:42.013989 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:41.996723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:52765 (local address 127.22.221.126:41743)
0926 07:33:41.996914 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:41.996918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:41.998026 (+ 1108us) server_negotiation.cc:407] Connection header received
0926 07:33:41.999959 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:41.999965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.000056 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.000167 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.001660 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.002386 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.004078 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.004557 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.004689 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.008186 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.008190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.008194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.008230 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.010927 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.011730 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.011737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.011745 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.011876 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.012308 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.012315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.012323 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.012570 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.012727 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.013427 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.013647 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:33:42.015033 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:41.997181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:41743 (local address 127.22.221.66:52765)
0926 07:33:41.997863 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:41.997907 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:41.999654 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:42.000400 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:42.000409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:42.000942 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:42.001487 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:42.001503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.002535 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.002542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:42.003921 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:42.003939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.004795 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.004803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:42.005025 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.005677 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:42.005712 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:42.007958 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:42.011075 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:42.011084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:42.011100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:42.011596 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:42.011988 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:42.011995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:42.011998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:42.012187 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:42.012700 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:42.012715 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:42.013060 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 07:33:42.013326 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 07:33:42.013850 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":260,"threads_started":1}
I20260926 07:33:42.016518 27337 heartbeater.cc:347] Connected to a master server at 127.22.221.126:41743
I20260926 07:33:42.016852 27337 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:42.017717 27337 heartbeater.cc:511] Master 127.22.221.126:41743 requested a full tablet report, sending...
I20260926 07:33:42.019322 27014 ts_manager.cc:194] Registered new tserver with Master: 2ef274a0583f4db689f85292e8915028 (127.22.221.66:34653)
I20260926 07:33:42.020140 27014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:52765
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:42.050093 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:41743
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:42.368690 27342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:42.368947 27342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:42.369038 27342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:42.376302 27342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:42.376400 27342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:42.376564 27342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:42.385979 27342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:41743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:42.387221 27342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:42.388522 27342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:42.401619 27350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:42.401661 27347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:42.401935 27348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:42.402357 27342 server_base.cc:1034] running on GCE node
I20260926 07:33:42.402949 27342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:42.403950 27342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:42.405141 27342 hybrid_clock.cc:648] HybridClock initialized: now 1790408022405090 us; error 55 us; skew 500 ppm
I20260926 07:33:42.405417 27342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:42.408795 27342 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:42.410140 27342 webserver.cc:460] Webserver started at http://127.22.221.67:44109/ using document root <none> and password file <none>
I20260926 07:33:42.410791 27342 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:42.410925 27342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:42.411283 27342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:42.414031 27342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "7e1534a828964e28afcdc60268456822"
format_stamp: "Formatted at 2026-09-26 07:33:42 on dist-test-slave-qlq8"
server_key: "79bf5f38829a33973a36513641ae607b"
server_key_iv: "6a0ee53a6414b6352bcf180319c54b11"
server_key_version: "encryptionkey@0"
I20260926 07:33:42.414791 27342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "7e1534a828964e28afcdc60268456822"
format_stamp: "Formatted at 2026-09-26 07:33:42 on dist-test-slave-qlq8"
server_key: "79bf5f38829a33973a36513641ae607b"
server_key_iv: "6a0ee53a6414b6352bcf180319c54b11"
server_key_version: "encryptionkey@0"
I20260926 07:33:42.420264 27342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 07:33:42.423678 27357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:42.424907 27342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:33:42.425089 27342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "7e1534a828964e28afcdc60268456822"
format_stamp: "Formatted at 2026-09-26 07:33:42 on dist-test-slave-qlq8"
server_key: "79bf5f38829a33973a36513641ae607b"
server_key_iv: "6a0ee53a6414b6352bcf180319c54b11"
server_key_version: "encryptionkey@0"
I20260926 07:33:42.425293 27342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:42.451367 27342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:42.455276 27342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:42.455564 27342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:42.456610 27342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:42.458077 27342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:42.458180 27342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:42.458305 27342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:42.458387 27342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:42.496011 27342 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:46581
I20260926 07:33:42.496030 27469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:46581 every 8 connection(s)
I20260926 07:33:42.497439 27342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:42.498613 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27342
I20260926 07:33:42.498828 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:42.499409 23413 external_mini_cluster.cc:1428] Setting key 53957512a8b019bd101c7b1c6b844a51
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:42.518419 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.500157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:45943 (local address 127.22.221.126:41743)
0926 07:33:42.500362 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:42.500368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:42.501713 (+ 1345us) server_negotiation.cc:407] Connection header received
0926 07:33:42.503779 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:42.503784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.503899 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.504009 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.505429 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.506176 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.507755 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.508162 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.508299 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.511619 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.511628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.511633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.511671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.515161 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.516012 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.516017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.516020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.516153 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.516666 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.516671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.516674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.517006 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.517165 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.517833 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.518041 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:33:42.519346 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.500768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:41743 (local address 127.22.221.67:45943)
0926 07:33:42.501561 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:42.501609 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:42.503528 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:42.504235 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:42.504243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:42.504739 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:42.505273 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:42.505289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.506327 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.506332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:42.507602 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:42.507618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.508394 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.508400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:42.508631 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.509292 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:42.509313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:42.511428 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:42.515432 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:42.515439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:42.515448 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:42.515879 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:42.516276 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:42.516282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:42.516286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:42.516537 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:42.517139 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:42.517147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:42.517476 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 07:33:42.517726 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 07:33:42.518209 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":294,"threads_started":1}
I20260926 07:33:42.520701 27470 heartbeater.cc:347] Connected to a master server at 127.22.221.126:41743
I20260926 07:33:42.520996 27470 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:42.521644 27470 heartbeater.cc:511] Master 127.22.221.126:41743 requested a full tablet report, sending...
I20260926 07:33:42.522933 27014 ts_manager.cc:194] Registered new tserver with Master: 7e1534a828964e28afcdc60268456822 (127.22.221.67:46581)
I20260926 07:33:42.523756 27014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:45943
I20260926 07:33:42.526865 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:42.543568 27203 heartbeater.cc:503] Master 127.22.221.126:41743 was elected leader, sending a full tablet report...
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:42.574389 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.559718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.22.221.65:38133)
0926 07:33:42.560297 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:42.560308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:42.560399 (+ 91us) server_negotiation.cc:407] Connection header received
0926 07:33:42.560603 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:42.560611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.560824 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.561005 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.562374 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.563481 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.564709 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.565252 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.565397 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.569288 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.569298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.569311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.569346 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.571961 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.572605 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.572612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.572615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.572746 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.573071 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.573074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.573076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.573453 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.573664 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.573926 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.574113 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":268,"threads_started":1}
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:42.590445 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.578411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.22.221.126:41743)
0926 07:33:42.578624 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:42.578630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:42.578784 (+ 154us) server_negotiation.cc:407] Connection header received
0926 07:33:42.578903 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:42.578907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.578966 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.579043 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.580210 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.580895 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.582035 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.582407 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.582499 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.585974 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.585979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.585982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.586012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.588292 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.588860 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.588863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.588865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.588994 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.589279 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.589282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.589283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.589553 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.589776 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.590009 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.590223 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:33:42.590968 27014 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37424
I20260926 07:33:42.591917 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27075
I20260926 07:33:42.603368 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.592715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.22.221.65:38133)
0926 07:33:42.592858 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:42.592862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:42.593159 (+ 297us) server_negotiation.cc:407] Connection header received
0926 07:33:42.593261 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:42.593264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.593349 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.593426 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.594472 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.595109 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.596550 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.596932 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.597019 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.599028 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.599032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.599035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.599058 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.601288 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.601820 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.601824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.601826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.601933 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.602230 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.602233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.602235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.602444 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.602604 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.602819 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.602970 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:33:42.603873 27198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:42.685554 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27075
I20260926 07:33:42.701478 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27209
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:42.716821 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.702326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.22.221.66:34653)
0926 07:33:42.702886 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:42.702892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:42.702917 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:33:42.702978 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:42.702984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.703205 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.703416 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.704540 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.705550 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.706816 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.707465 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.707565 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.711583 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.711594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.711608 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.711638 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.714216 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.714822 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.714829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.714831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.714940 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.715402 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.715405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.715407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.715717 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.715945 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.716218 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.716400 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260926 07:33:42.718104 27332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:42.799690 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27209
I20260926 07:33:42.815446 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27342
Sep 26 07:33:42 dist-test-slave-qlq8 krb5kdc[26969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:42.831378 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:42.816286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.22.221.67:46581)
0926 07:33:42.816937 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:42.816944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:42.817005 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:33:42.817089 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:42.817097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:42.817399 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:42.817533 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:42.818793 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.819978 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.821254 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:42.821789 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:42.821903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:42.826095 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:42.826106 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:42.826119 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:42.826153 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:42.828913 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.829514 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.829521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.829523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.829643 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:42.829944 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:42.829947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:42.829949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:42.830330 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:42.830561 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:42.830803 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:33:42.831021 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":326,"threads_started":1}
I20260926 07:33:42.832608 27465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:42.914484 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27342
I20260926 07:33:42.930410 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 26982
I20260926 07:33:42.931643 27044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:43.033927 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 26982
2026-09-26T07:33:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3848 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:43 dist-test-slave-qlq8 krb5kdc[27489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:43 dist-test-slave-qlq8 krb5kdc[27489](info): set up 2 sockets
Sep 26 07:33:43 dist-test-slave-qlq8 krb5kdc[27489](info): commencing operation
krb5kdc: starting...
W20260926 07:33:44.123898 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:44 dist-test-slave-qlq8 krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:44.320768 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35759
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:44.635294 27502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:44.635607 27502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:44.635782 27502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:44.642731 27502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:44.642876 27502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:44.642992 27502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:44.643097 27502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:44.643172 27502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:44.653126 27502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:44.654361 27502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:44.655651 27502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:44.667023 27508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:44.667165 27507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:44.667361 27510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:44.668027 27502 server_base.cc:1034] running on GCE node
I20260926 07:33:44.668673 27502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:44.669879 27502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:44.671108 27502 hybrid_clock.cc:648] HybridClock initialized: now 1790408024671077 us; error 39 us; skew 500 ppm
I20260926 07:33:44.671478 27502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:44 dist-test-slave-qlq8 krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408024, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:44.674405 27502 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:44.675616 27502 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:44.675741 27502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:44.676059 27502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:44.678849 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "6363d080aa554068bb381407f58f8ec2"
format_stamp: "Formatted at 2026-09-26 07:33:44 on dist-test-slave-qlq8"
server_key: "b5404dd08611ce5b27c9f8a5e550401b"
server_key_iv: "4dac4ad47a7cf7aaa46a49556f3d5a25"
server_key_version: "encryptionkey@0"
I20260926 07:33:44.679679 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "6363d080aa554068bb381407f58f8ec2"
format_stamp: "Formatted at 2026-09-26 07:33:44 on dist-test-slave-qlq8"
server_key: "b5404dd08611ce5b27c9f8a5e550401b"
server_key_iv: "4dac4ad47a7cf7aaa46a49556f3d5a25"
server_key_version: "encryptionkey@0"
I20260926 07:33:44.684520 27502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:33:44.687852 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:44.689090 27502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:44.689278 27502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "6363d080aa554068bb381407f58f8ec2"
format_stamp: "Formatted at 2026-09-26 07:33:44 on dist-test-slave-qlq8"
server_key: "b5404dd08611ce5b27c9f8a5e550401b"
server_key_iv: "4dac4ad47a7cf7aaa46a49556f3d5a25"
server_key_version: "encryptionkey@0"
I20260926 07:33:44.689493 27502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:44.711478 27502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:44.715147 27502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:44.715440 27502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:44.733199 27502 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:35759
I20260926 07:33:44.733215 27567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:35759 every 8 connection(s)
I20260926 07:33:44.734058 27502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:44.736970 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27502
I20260926 07:33:44.737207 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:44.737869 27568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:44.738029 23413 external_mini_cluster.cc:1428] Setting key 9f6a67faac3be4710de3d28fcf7a6a31
I20260926 07:33:44.752076 27568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2: Bootstrap starting.
Sep 26 07:33:44 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:44.755070 27568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:44.756119 27568 log.cc:824] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:44.758636 27568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2: No bootstrap required, opened a new log
I20260926 07:33:44.760337 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:44.740069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.22.221.126:35759)
0926 07:33:44.740883 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:44.740893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:44.740935 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:33:44.742490 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:44.742516 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:44.742864 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:44.743169 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:44.745017 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:44.746260 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:44.748044 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:44.748708 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:44.748849 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:44.753428 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:44.753436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:44.753448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:44.753489 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:44.756516 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:44.757192 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:44.757198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:44.757204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:44.757377 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:44.757778 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:44.757782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:44.757784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:44.758247 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:44.758569 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:44.758955 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 07:33:44.759404 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":376,"threads_started":1}
I20260926 07:33:44.766561 27568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6363d080aa554068bb381407f58f8ec2" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35759 } }
I20260926 07:33:44.766929 27568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:44.767035 27568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6363d080aa554068bb381407f58f8ec2, State: Initialized, Role: FOLLOWER
I20260926 07:33:44.767618 27568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6363d080aa554068bb381407f58f8ec2" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35759 } }
I20260926 07:33:44.767876 27568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:44.768013 27568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:44.768193 27568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:44.770278 27568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6363d080aa554068bb381407f58f8ec2" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35759 } }
I20260926 07:33:44.770756 27568 leader_election.cc:302] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6363d080aa554068bb381407f58f8ec2; no voters:
I20260926 07:33:44.771514 27568 leader_election.cc:288] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:44.771607 27573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:44.772404 27573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 6363d080aa554068bb381407f58f8ec2, State: Running, Role: LEADER
I20260926 07:33:44.773087 27573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6363d080aa554068bb381407f58f8ec2" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35759 } }
I20260926 07:33:44.773882 27568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:44.776149 27574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6363d080aa554068bb381407f58f8ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6363d080aa554068bb381407f58f8ec2" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35759 } } }
I20260926 07:33:44.776126 27575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6363d080aa554068bb381407f58f8ec2. Latest consensus state: current_term: 1 leader_uuid: "6363d080aa554068bb381407f58f8ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6363d080aa554068bb381407f58f8ec2" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35759 } } }
I20260926 07:33:44.776531 27575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:44.776527 27574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:44.783993 27579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:44.789393 27579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:44.798660 27579 catalog_manager.cc:1347] Generated new cluster ID: 6fdb4576a9d5488e881498e7a0496b27
I20260926 07:33:44.798774 27579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:44.812795 27579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:44.814594 27579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:44.821522 27579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2: Generated new TSK 0
I20260926 07:33:44.822536 27579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:44.868057 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35759
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:45.185491 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:45.185747 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:45.185842 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:45.193084 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:45.193178 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:45.193289 27594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:45.202788 27594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:45.204070 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:45.205281 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:45.218081 27599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.218030 27600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.218128 27602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:45.218946 27594 server_base.cc:1034] running on GCE node
I20260926 07:33:45.219585 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:45.220468 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:45.221701 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790408025221657 us; error 58 us; skew 500 ppm
I20260926 07:33:45.222011 27594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:45 dist-test-slave-qlq8 krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:45.225364 27594 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:45.226457 27594 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:45.226593 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:45.226899 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:45.229769 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "ad450b96736a485594e1912a30c7c9f6"
format_stamp: "Formatted at 2026-09-26 07:33:45 on dist-test-slave-qlq8"
server_key: "cb56b0668413653f7a8f8beda9c2298c"
server_key_iv: "54f4320a84c325030a191d5f16f46690"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.230566 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "ad450b96736a485594e1912a30c7c9f6"
format_stamp: "Formatted at 2026-09-26 07:33:45 on dist-test-slave-qlq8"
server_key: "cb56b0668413653f7a8f8beda9c2298c"
server_key_iv: "54f4320a84c325030a191d5f16f46690"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.235586 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:33:45.238824 27608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.240150 27594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:45.240337 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "ad450b96736a485594e1912a30c7c9f6"
format_stamp: "Formatted at 2026-09-26 07:33:45 on dist-test-slave-qlq8"
server_key: "cb56b0668413653f7a8f8beda9c2298c"
server_key_iv: "54f4320a84c325030a191d5f16f46690"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.240552 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:45.258610 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:45.262887 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:45.263157 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:45.264110 27594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:45.265553 27594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:45.265689 27594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.265825 27594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:45.265914 27594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.300025 27594 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:40247
I20260926 07:33:45.300068 27720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:40247 every 8 connection(s)
I20260926 07:33:45.300889 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:45.305544 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27594
I20260926 07:33:45.305787 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:45.306564 23413 external_mini_cluster.cc:1428] Setting key e17c9a4cae394f1550a5a1c783e803a6
Sep 26 07:33:45 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:45.322911 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.303271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:43437 (local address 127.22.221.126:35759)
0926 07:33:45.303734 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:45.303738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:45.304522 (+ 784us) server_negotiation.cc:407] Connection header received
0926 07:33:45.306921 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:45.306926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:45.306998 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:45.307092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:45.308905 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.309621 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.312111 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.312489 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.312581 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:45.316746 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:45.316751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:45.316754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:45.316785 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:45.319420 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:45.320320 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:45.320324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:45.320327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:45.320437 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:45.320945 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:45.320949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:45.320951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:45.321353 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:45.321468 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:45.322356 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 07:33:45.322580 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":258,"threads_started":1}
I20260926 07:33:45.324321 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.303710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35759 (local address 127.22.221.65:43437)
0926 07:33:45.304370 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.304411 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:45.306619 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:45.307378 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:45.307386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:45.308017 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:45.308733 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:45.308757 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.309778 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.309787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:45.311971 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:45.311989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.312706 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.312716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:45.313043 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:45.313783 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:45.313815 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:45.316533 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:45.319546 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:45.319558 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:45.319575 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:45.320204 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:45.320547 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:45.320555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:45.320561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:45.320820 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:45.321494 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:45.321505 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:45.321958 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 07:33:45.322263 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 07:33:45.322838 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":259,"threads_started":1}
I20260926 07:33:45.325982 27721 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35759
I20260926 07:33:45.326295 27721 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:45.326928 27721 heartbeater.cc:511] Master 127.22.221.126:35759 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:45.329039 27533 ts_manager.cc:194] Registered new tserver with Master: ad450b96736a485594e1912a30c7c9f6 (127.22.221.65:40247)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:33:45.330722 27533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:43437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:45.354528 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35759
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:45.674647 27727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:45.674953 27727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:45.675148 27727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:45.682142 27727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:45.682266 27727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:45.682451 27727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:45.691854 27727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:45.693141 27727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:45.694420 27727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:45.706988 27732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.707075 27733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.707059 27735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:45.707914 27727 server_base.cc:1034] running on GCE node
I20260926 07:33:45.708503 27727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:45.709362 27727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:45.710737 27727 hybrid_clock.cc:648] HybridClock initialized: now 1790408025710664 us; error 53 us; skew 500 ppm
I20260926 07:33:45.711046 27727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:45 dist-test-slave-qlq8 krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:45.713984 27727 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:45.715039 27727 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:45.715163 27727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:45.715490 27727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:45.718227 27727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "6685cdb5d447490fb8e4ad68912fc2be"
format_stamp: "Formatted at 2026-09-26 07:33:45 on dist-test-slave-qlq8"
server_key: "89940b0124e1a97c871f679bcbc92fbb"
server_key_iv: "ca4eda535a2913f13dc6165fcafd8495"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.718989 27727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "6685cdb5d447490fb8e4ad68912fc2be"
format_stamp: "Formatted at 2026-09-26 07:33:45 on dist-test-slave-qlq8"
server_key: "89940b0124e1a97c871f679bcbc92fbb"
server_key_iv: "ca4eda535a2913f13dc6165fcafd8495"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.723860 27727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:33:45.727025 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.728551 27727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:45.728680 27727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "6685cdb5d447490fb8e4ad68912fc2be"
format_stamp: "Formatted at 2026-09-26 07:33:45 on dist-test-slave-qlq8"
server_key: "89940b0124e1a97c871f679bcbc92fbb"
server_key_iv: "ca4eda535a2913f13dc6165fcafd8495"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.728847 27727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:45.747673 27727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:45.751296 27727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:45.751513 27727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:45.752331 27727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:45.753535 27727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:45.753618 27727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.753700 27727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:45.753754 27727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.786293 27727 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:39111
I20260926 07:33:45.786334 27853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:39111 every 8 connection(s)
I20260926 07:33:45.787099 27727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:45.791026 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27727
I20260926 07:33:45.791222 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:45.791812 23413 external_mini_cluster.cc:1428] Setting key a3be212b0ecb8356ad354db1e1e30591
Sep 26 07:33:45 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:45.808847 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.789503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:60213 (local address 127.22.221.126:35759)
0926 07:33:45.789716 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:45.789722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:45.790963 (+ 1241us) server_negotiation.cc:407] Connection header received
0926 07:33:45.793176 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:45.793182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:45.793253 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:45.793353 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:45.795206 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.795982 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.798465 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.799096 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.799216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:45.803327 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:45.803332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:45.803335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:45.803364 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:45.805853 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:45.806562 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:45.806571 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:45.806575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:45.806711 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:45.807127 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:45.807137 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:45.807139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:45.807451 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:45.807618 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:45.808304 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 07:33:45.808522 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:33:45.809759 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.790002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35759 (local address 127.22.221.66:60213)
0926 07:33:45.790753 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.790816 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:45.792950 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:45.793538 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:45.793545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:45.794249 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:45.795027 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:45.795052 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.796138 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.796146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:45.798306 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:45.798323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:45.799348 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:45.799357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:45.799657 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:45.800466 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:45.800501 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:45.803098 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:45.806007 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:45.806016 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:45.806031 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:45.806439 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:45.806823 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:45.806829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:45.806832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:45.807003 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:45.807580 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:45.807588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:45.807927 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 07:33:45.808188 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 07:33:45.808692 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:33:45.811173 27854 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35759
I20260926 07:33:45.811532 27854 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:45.812153 27854 heartbeater.cc:511] Master 127.22.221.126:35759 requested a full tablet report, sending...
I20260926 07:33:45.813560 27533 ts_manager.cc:194] Registered new tserver with Master: 6685cdb5d447490fb8e4ad68912fc2be (127.22.221.66:39111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
I20260926 07:33:45.814558 27533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:60213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:45.838625 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35759
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:46.146406 27859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:46.146669 27859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:46.146844 27859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:46.153966 27859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:46.154096 27859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:46.154291 27859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:46.163692 27859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:46.164945 27859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:46.166234 27859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:46.178689 27864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:46.178787 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:46.178746 27867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:46.179489 27859 server_base.cc:1034] running on GCE node
I20260926 07:33:46.180127 27859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:46.180950 27859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:46.182204 27859 hybrid_clock.cc:648] HybridClock initialized: now 1790408026182142 us; error 57 us; skew 500 ppm
I20260926 07:33:46.182478 27859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:46 dist-test-slave-qlq8 krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408026, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:46.190400 27859 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:46.191634 27859 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:46.191746 27859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:46.192066 27859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:46.194942 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "7e1cde9fb20b4626a54fda98a904a92f"
format_stamp: "Formatted at 2026-09-26 07:33:46 on dist-test-slave-qlq8"
server_key: "68afaf29686b3a293e5dc18fd062c096"
server_key_iv: "e6249fc20d22f348e4d2682614443038"
server_key_version: "encryptionkey@0"
I20260926 07:33:46.195780 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "7e1cde9fb20b4626a54fda98a904a92f"
format_stamp: "Formatted at 2026-09-26 07:33:46 on dist-test-slave-qlq8"
server_key: "68afaf29686b3a293e5dc18fd062c096"
server_key_iv: "e6249fc20d22f348e4d2682614443038"
server_key_version: "encryptionkey@0"
I20260926 07:33:46.200503 27859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 07:33:46.203817 27873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:46.205179 27859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:46.205358 27859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "7e1cde9fb20b4626a54fda98a904a92f"
format_stamp: "Formatted at 2026-09-26 07:33:46 on dist-test-slave-qlq8"
server_key: "68afaf29686b3a293e5dc18fd062c096"
server_key_iv: "e6249fc20d22f348e4d2682614443038"
server_key_version: "encryptionkey@0"
I20260926 07:33:46.205582 27859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:46.229192 27859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:46.233611 27859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:46.233915 27859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:46.234859 27859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:46.236146 27859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:46.236265 27859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:46.236404 27859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:46.236498 27859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:46.268824 27859 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:34351
I20260926 07:33:46.268909 27985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:34351 every 8 connection(s)
I20260926 07:33:46.269675 27859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:46.275589 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 27859
I20260926 07:33:46.275794 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:46.276225 23413 external_mini_cluster.cc:1428] Setting key 42858503424110031477eba5fa48eabc
Sep 26 07:33:46 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408026, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:46.288883 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.272050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:33155 (local address 127.22.221.126:35759)
0926 07:33:46.272219 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.272223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.273279 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 07:33:46.275055 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.275060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.275130 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.275362 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:46.277168 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.278084 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.279741 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.280136 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.280240 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.283455 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:46.283460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:46.283464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:46.283495 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:46.286187 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.286882 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.286887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.286889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.287000 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.287436 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.287439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.287441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.287706 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:46.287820 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.288492 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.288642 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:33:46.290156 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.272449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35759 (local address 127.22.221.67:33155)
0926 07:33:46.273111 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.273146 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:46.274810 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:46.275534 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:46.275541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:46.276147 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:46.276811 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.276835 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.278238 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.278246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.279613 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.279624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.280314 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.280318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.280490 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.281171 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:46.281192 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:46.283218 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:46.286334 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:46.286340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:46.286351 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:46.286780 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:46.287098 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:46.287101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:46.287103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:46.287341 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:46.287854 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:46.287860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:46.288174 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 07:33:46.288418 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 07:33:46.288878 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":263,"threads_started":1}
I20260926 07:33:46.291651 27986 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35759
I20260926 07:33:46.291949 27986 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:46.292621 27986 heartbeater.cc:511] Master 127.22.221.126:35759 requested a full tablet report, sending...
I20260926 07:33:46.294103 27532 ts_manager.cc:194] Registered new tserver with Master: 7e1cde9fb20b4626a54fda98a904a92f (127.22.221.67:34351)
I20260926 07:33:46.295037 27532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:33155
I20260926 07:33:46.303994 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:46.320943 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.308382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.22.221.126:35759)
0926 07:33:46.308546 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.308550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.309093 (+ 543us) server_negotiation.cc:407] Connection header received
0926 07:33:46.309285 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.309291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.309372 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.309476 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:46.310940 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.311722 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.313007 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.313456 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.313597 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.315987 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:46.315992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:46.315996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:46.316025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:46.318352 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.318998 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.319006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.319009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.319143 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.319535 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.319543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.319549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.319854 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:46.320105 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.320434 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.320648 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:33:46.324972 27532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:46.332273 27532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:46.334153 27721 heartbeater.cc:503] Master 127.22.221.126:35759 was elected leader, sending a full tablet report...
I20260926 07:33:46.355932 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.344371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.22.221.65:40247)
0926 07:33:46.345136 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.345144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.345171 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:33:46.345466 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.345476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.345726 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.345880 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:46.346904 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.348760 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.352168 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.354086 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.354231 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.354528 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.355351 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.355580 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":1446,"threads_started":1}
I20260926 07:33:46.357270 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.344152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40247 (local address 127.0.0.1:59804)
0926 07:33:46.344970 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.345010 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:46.345179 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:46.346221 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:46.346230 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:46.346269 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:46.346692 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.346710 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.348995 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.349003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.351993 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.352007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.354754 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.354763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.355027 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.355094 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 07:33:46.356408 (+ 1314us) client_negotiation.cc:240] Negotiation successful
0926 07:33:46.356690 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1,"thread_start_us":294,"threads_started":1}
I20260926 07:33:46.357977 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.345449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.22.221.67:34351)
0926 07:33:46.346432 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.346439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.346472 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:33:46.346585 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.346595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.346856 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.348031 (+ 1175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:46.348420 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.350526 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.353989 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.355974 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.356143 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.356433 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.357386 (+ 953us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.357594 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"spinlock_wait_cycles":18304,"thread_start_us":799,"threads_started":1}
I20260926 07:33:46.358098 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.344818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:34351 (local address 127.0.0.1:34092)
0926 07:33:46.345584 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.345604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:46.345708 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:46.347224 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:46.347251 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:46.347287 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:46.347810 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.347826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.350732 (+ 2906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.350740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.353743 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.353764 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.356619 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.356629 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.356882 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.356934 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:33:46.357043 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:33:46.357252 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1,"thread_start_us":938,"threads_started":1}
I20260926 07:33:46.361872 27656 tablet_service.cc:1467] Processing CreateTablet for tablet a5ab4b9a3de745d78c3a39df6851bf9d (DEFAULT_TABLE table=test-table [id=f792cf586a444090896d542cf0b38f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:46.362046 27921 tablet_service.cc:1467] Processing CreateTablet for tablet a5ab4b9a3de745d78c3a39df6851bf9d (DEFAULT_TABLE table=test-table [id=f792cf586a444090896d542cf0b38f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:46.363271 27921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ab4b9a3de745d78c3a39df6851bf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:46.363278 27656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ab4b9a3de745d78c3a39df6851bf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:46.367499 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.346138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39111 (local address 127.0.0.1:60580)
0926 07:33:46.357910 (+ 11772us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.357932 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:46.358056 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:46.359481 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:46.359485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:46.359510 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:46.360810 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.360823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.362722 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.362727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.364692 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.364706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.366714 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.366721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.366894 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.366928 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:33:46.367022 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:33:46.367143 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11649,"thread_start_us":12003,"threads_started":1}
I20260926 07:33:46.367769 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.346342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.22.221.66:39111)
0926 07:33:46.358896 (+ 12554us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.358905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.358931 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:33:46.359022 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.359031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.359299 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.359494 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:46.361004 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.362584 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.364860 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.366253 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.366394 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.366562 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.367304 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.367482 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12402,"thread_start_us":12300,"threads_started":1}
I20260926 07:33:46.371403 27789 tablet_service.cc:1467] Processing CreateTablet for tablet a5ab4b9a3de745d78c3a39df6851bf9d (DEFAULT_TABLE table=test-table [id=f792cf586a444090896d542cf0b38f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:46.372747 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ab4b9a3de745d78c3a39df6851bf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:46.375788 28004 tablet_bootstrap.cc:492] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Bootstrap starting.
I20260926 07:33:46.377645 28005 tablet_bootstrap.cc:492] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Bootstrap starting.
I20260926 07:33:46.379372 28004 tablet_bootstrap.cc:654] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:46.380605 28004 log.cc:824] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:46.381330 28005 tablet_bootstrap.cc:654] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:46.382516 28005 log.cc:824] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:46.383634 28004 tablet_bootstrap.cc:492] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: No bootstrap required, opened a new log
I20260926 07:33:46.383882 28004 ts_tablet_manager.cc:1397] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260926 07:33:46.385233 28005 tablet_bootstrap.cc:492] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: No bootstrap required, opened a new log
I20260926 07:33:46.385469 28005 ts_tablet_manager.cc:1397] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:33:46.386282 28008 tablet_bootstrap.cc:492] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Bootstrap starting.
I20260926 07:33:46.389894 28008 tablet_bootstrap.cc:654] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:46.391034 28008 log.cc:824] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:46.391623 28004 raft_consensus.cc:348] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.392097 28004 raft_consensus.cc:374] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:46.392203 28004 raft_consensus.cc:729] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1cde9fb20b4626a54fda98a904a92f, State: Initialized, Role: FOLLOWER
I20260926 07:33:46.392899 28004 consensus_queue.cc:260] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.394125 27986 heartbeater.cc:503] Master 127.22.221.126:35759 was elected leader, sending a full tablet report...
I20260926 07:33:46.394239 28008 tablet_bootstrap.cc:492] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: No bootstrap required, opened a new log
I20260926 07:33:46.394541 28008 ts_tablet_manager.cc:1397] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:33:46.394830 28004 ts_tablet_manager.cc:1428] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20260926 07:33:46.397003 28005 raft_consensus.cc:348] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.397429 28005 raft_consensus.cc:374] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:46.397540 28005 raft_consensus.cc:729] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad450b96736a485594e1912a30c7c9f6, State: Initialized, Role: FOLLOWER
I20260926 07:33:46.398113 28005 consensus_queue.cc:260] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.399866 28005 ts_tablet_manager.cc:1428] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 07:33:46.403291 28008 raft_consensus.cc:348] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.403615 28008 raft_consensus.cc:374] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:46.403687 28008 raft_consensus.cc:729] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6685cdb5d447490fb8e4ad68912fc2be, State: Initialized, Role: FOLLOWER
I20260926 07:33:46.404187 28008 consensus_queue.cc:260] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.405134 27854 heartbeater.cc:503] Master 127.22.221.126:35759 was elected leader, sending a full tablet report...
I20260926 07:33:46.405702 28008 ts_tablet_manager.cc:1428] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 07:33:46.472640 28011 raft_consensus.cc:482] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:46.472923 28011 raft_consensus.cc:504] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.474659 28011 leader_election.cc:288] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e1cde9fb20b4626a54fda98a904a92f (127.22.221.67:34351), 6685cdb5d447490fb8e4ad68912fc2be (127.22.221.66:39111)
I20260926 07:33:46.484663 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.475098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:57025 (local address 127.22.221.67:34351)
0926 07:33:46.475353 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.475358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.475877 (+ 519us) server_negotiation.cc:407] Connection header received
0926 07:33:46.476519 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.476526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.476627 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.476729 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:46.477736 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.478871 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.482216 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.483529 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.483695 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.483792 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.484204 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.484346 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:33:46.484779 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.476337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:33379 (local address 127.22.221.66:39111)
0926 07:33:46.476529 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.476533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.476658 (+ 125us) server_negotiation.cc:407] Connection header received
0926 07:33:46.476813 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.476817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.476873 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.476966 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:46.477797 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.478873 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.482216 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.483650 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.483819 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.483909 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.484394 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.484518 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:33:46.484831 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.475041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:34351 (local address 127.22.221.65:57025)
0926 07:33:46.475736 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.475756 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:46.475894 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:46.476942 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:46.476950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:46.476994 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:46.477541 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.477567 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.479046 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.479050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.482045 (+ 2995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.482062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.483813 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.483818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.483956 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.484027 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 07:33:46.484175 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 07:33:46.484328 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":94,"thread_start_us":364,"threads_started":1}
I20260926 07:33:46.484894 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.475894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39111 (local address 127.22.221.65:33379)
0926 07:33:46.476534 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.476554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:46.476692 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:46.476989 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:46.476994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:46.477013 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:46.477617 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.477631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.479038 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.479045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.482045 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:46.482062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.483965 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.483972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:46.484128 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.484150 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:33:46.484334 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 07:33:46.484463 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":189,"thread_start_us":295,"threads_started":1}
I20260926 07:33:46.485896 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ab4b9a3de745d78c3a39df6851bf9d" candidate_uuid: "ad450b96736a485594e1912a30c7c9f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6685cdb5d447490fb8e4ad68912fc2be" is_pre_election: true
I20260926 07:33:46.486181 27941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ab4b9a3de745d78c3a39df6851bf9d" candidate_uuid: "ad450b96736a485594e1912a30c7c9f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1cde9fb20b4626a54fda98a904a92f" is_pre_election: true
I20260926 07:33:46.486392 27809 raft_consensus.cc:2454] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad450b96736a485594e1912a30c7c9f6 in term 0.
I20260926 07:33:46.486691 27941 raft_consensus.cc:2454] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad450b96736a485594e1912a30c7c9f6 in term 0.
I20260926 07:33:46.487069 27611 leader_election.cc:302] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6685cdb5d447490fb8e4ad68912fc2be, ad450b96736a485594e1912a30c7c9f6; no voters:
I20260926 07:33:46.487506 28011 raft_consensus.cc:2781] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:46.487682 28011 raft_consensus.cc:482] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:46.487818 28011 raft_consensus.cc:3037] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:46.490154 28011 raft_consensus.cc:504] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.490801 28011 leader_election.cc:288] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [CANDIDATE]: Term 1 election: Requested vote from peers 7e1cde9fb20b4626a54fda98a904a92f (127.22.221.67:34351), 6685cdb5d447490fb8e4ad68912fc2be (127.22.221.66:39111)
I20260926 07:33:46.491277 27941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ab4b9a3de745d78c3a39df6851bf9d" candidate_uuid: "ad450b96736a485594e1912a30c7c9f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1cde9fb20b4626a54fda98a904a92f"
I20260926 07:33:46.491293 27809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ab4b9a3de745d78c3a39df6851bf9d" candidate_uuid: "ad450b96736a485594e1912a30c7c9f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6685cdb5d447490fb8e4ad68912fc2be"
I20260926 07:33:46.491492 27809 raft_consensus.cc:3037] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:46.491503 27941 raft_consensus.cc:3037] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:46.494023 27941 raft_consensus.cc:2454] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad450b96736a485594e1912a30c7c9f6 in term 1.
I20260926 07:33:46.494078 27809 raft_consensus.cc:2454] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad450b96736a485594e1912a30c7c9f6 in term 1.
I20260926 07:33:46.494388 27611 leader_election.cc:302] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e1cde9fb20b4626a54fda98a904a92f, ad450b96736a485594e1912a30c7c9f6; no voters:
I20260926 07:33:46.494694 28011 raft_consensus.cc:2781] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:46.495188 28011 raft_consensus.cc:686] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 1 LEADER]: Becoming Leader. State: Replica: ad450b96736a485594e1912a30c7c9f6, State: Running, Role: LEADER
I20260926 07:33:46.495832 28011 consensus_queue.cc:237] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } }
I20260926 07:33:46.500697 27532 catalog_manager.cc:5495] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad450b96736a485594e1912a30c7c9f6 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "ad450b96736a485594e1912a30c7c9f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad450b96736a485594e1912a30c7c9f6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 } health_report { overall_health: UNKNOWN } } }
W20260926 07:33:46.521168 27987 tablet.cc:2369] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:33:46.539006 27855 tablet.cc:2369] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:46.544988 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.539906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.22.221.65:40247)
0926 07:33:46.540127 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.540132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.540150 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:33:46.540314 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.540318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.540402 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.540518 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:46.540822 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.541515 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.542765 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.543164 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.543343 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.543843 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:46.544344 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:46.544507 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.544606 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.544696 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:33:46.552572 27941 raft_consensus.cc:1261] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 1 FOLLOWER]: Refusing update from remote peer ad450b96736a485594e1912a30c7c9f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:46.552662 27809 raft_consensus.cc:1261] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 1 FOLLOWER]: Refusing update from remote peer ad450b96736a485594e1912a30c7c9f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 07:33:46.553033 27722 tablet.cc:2369] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:46.553669 28015 consensus_queue.cc:1035] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6685cdb5d447490fb8e4ad68912fc2be" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:46.554001 28011 consensus_queue.cc:1035] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e1cde9fb20b4626a54fda98a904a92f" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:46.564499 28020 mvcc.cc:205] Tried to move back new op lower bound from 7333511276750360576 to 7333511276531298304. Current Snapshot: MvccSnapshot[applied={T|T < 7333511276750360576}]
I20260926 07:33:46.578137 27533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:46.578380 27533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:46.581568 27533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6363d080aa554068bb381407f58f8ec2: Sending DeleteTablet for 3 replicas of tablet a5ab4b9a3de745d78c3a39df6851bf9d
I20260926 07:33:46.582536 27656 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ab4b9a3de745d78c3a39df6851bf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:46 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:59804
I20260926 07:33:46.582830 27921 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ab4b9a3de745d78c3a39df6851bf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:46 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:34092
I20260926 07:33:46.582971 27789 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ab4b9a3de745d78c3a39df6851bf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:46 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:60580
I20260926 07:33:46.584865 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27594
I20260926 07:33:46.594210 28028 tablet_replica.cc:330] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: stopping tablet replica
I20260926 07:33:46.594841 28028 raft_consensus.cc:2229] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:46.595705 28028 raft_consensus.cc:2258] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:46.597852 28028 ts_tablet_manager.cc:1905] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:46.598412 28029 tablet_replica.cc:330] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: stopping tablet replica
I20260926 07:33:46.598795 28029 raft_consensus.cc:2229] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:46.599329 28029 raft_consensus.cc:2258] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:46.603551 28029 ts_tablet_manager.cc:1905] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:46.606544 28028 ts_tablet_manager.cc:1918] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:46.606769 28028 log.cc:1196] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/a5ab4b9a3de745d78c3a39df6851bf9d
I20260926 07:33:46.607620 28028 ts_tablet_manager.cc:1939] T a5ab4b9a3de745d78c3a39df6851bf9d P ad450b96736a485594e1912a30c7c9f6: Deleting consensus metadata
I20260926 07:33:46.607923 28030 tablet_replica.cc:330] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: stopping tablet replica
I20260926 07:33:46.608354 28030 raft_consensus.cc:2229] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:46.608886 28030 raft_consensus.cc:2258] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:46.610380 27520 catalog_manager.cc:4841] TS ad450b96736a485594e1912a30c7c9f6 (127.22.221.65:40247): tablet a5ab4b9a3de745d78c3a39df6851bf9d (table test-table [id=f792cf586a444090896d542cf0b38f85]) successfully deleted
I20260926 07:33:46.610572 28029 ts_tablet_manager.cc:1918] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:46.610737 28029 log.cc:1196] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/a5ab4b9a3de745d78c3a39df6851bf9d
I20260926 07:33:46.611346 28029 ts_tablet_manager.cc:1939] T a5ab4b9a3de745d78c3a39df6851bf9d P 7e1cde9fb20b4626a54fda98a904a92f: Deleting consensus metadata
I20260926 07:33:46.612792 28030 ts_tablet_manager.cc:1905] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:46.613008 27519 catalog_manager.cc:4841] TS 7e1cde9fb20b4626a54fda98a904a92f (127.22.221.67:34351): tablet a5ab4b9a3de745d78c3a39df6851bf9d (table test-table [id=f792cf586a444090896d542cf0b38f85]) successfully deleted
I20260926 07:33:46.617638 28030 ts_tablet_manager.cc:1918] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:46.617815 28030 log.cc:1196] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/a5ab4b9a3de745d78c3a39df6851bf9d
I20260926 07:33:46.618340 28030 ts_tablet_manager.cc:1939] T a5ab4b9a3de745d78c3a39df6851bf9d P 6685cdb5d447490fb8e4ad68912fc2be: Deleting consensus metadata
Sep 26 07:33:46 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:46.619771 27519 catalog_manager.cc:4841] TS 6685cdb5d447490fb8e4ad68912fc2be (127.22.221.66:39111): tablet a5ab4b9a3de745d78c3a39df6851bf9d (table test-table [id=f792cf586a444090896d542cf0b38f85]) successfully deleted
I20260926 07:33:46.625768 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.586413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.22.221.65:40247)
0926 07:33:46.586621 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.586629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.601845 (+ 15216us) server_negotiation.cc:407] Connection header received
0926 07:33:46.602266 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.602271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.602402 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.602480 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:46.604297 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.605558 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.609269 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.609959 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.610064 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.619800 (+ 9736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:46.619815 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:46.619831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:46.619877 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:46.622779 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.623571 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.623577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.623579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.623723 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.624125 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.624128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.624130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.624473 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:46.624693 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.625017 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.625384 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:33:46.626421 27716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:46.724395 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27594
I20260926 07:33:46.741837 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27727
Sep 26 07:33:46 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:46.756897 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.742720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.22.221.66:39111)
0926 07:33:46.742920 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.742927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.742948 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:33:46.743010 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.743015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.743091 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.743192 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:46.744443 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.745615 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.746803 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.747525 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.747651 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.751499 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:46.751511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:46.751526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:46.751562 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:46.754402 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.754984 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.754991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.754993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.755113 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.755610 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.755613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.755615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.755963 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:46.756170 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.756469 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.756629 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:33:46.757598 27849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:46.866891 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27727
I20260926 07:33:46.883208 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27859
Sep 26 07:33:46 dist-test-slave-qlq8 krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:46.897864 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.884049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.22.221.67:34351)
0926 07:33:46.884273 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:46.884278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:46.884296 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:33:46.884456 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:46.884460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:46.884523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:46.884605 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:46.885896 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.886626 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.887870 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:46.888252 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:46.888345 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:46.892602 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:46.892616 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:46.892633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:46.892672 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:46.895554 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.896134 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.896140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.896142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.896281 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:46.896588 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:46.896591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:46.896593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:46.896908 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:46.897120 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:46.897340 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 07:33:46.897492 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 07:33:46.898486 27981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:46.986315 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27859
I20260926 07:33:47.004401 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 27502
I20260926 07:33:47.005498 27563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:47.114907 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 27502
2026-09-26T07:33:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4080 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T07:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:47Z Disabled control of system clock
I20260926 07:33:47.179380 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42969
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:42969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:33:47.488821 28039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:47.489099 28039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:47.489192 28039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:47.496337 28039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:47.496444 28039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:47.496505 28039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:47.496570 28039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:47.496608 28039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:47.505894 28039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:42969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:47.507110 28039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:47.508882 28039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:47.520171 28044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:47.520358 28045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:47.520398 28047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:47.521425 28039 server_base.cc:1034] running on GCE node
I20260926 07:33:47.522184 28039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:47.523562 28039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:47.524811 28039 hybrid_clock.cc:648] HybridClock initialized: now 1790408027524768 us; error 55 us; skew 500 ppm
I20260926 07:33:47.525166 28039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:47.527655 28039 webserver.cc:460] Webserver started at http://127.22.221.126:44809/ using document root <none> and password file <none>
I20260926 07:33:47.528334 28039 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:47.528474 28039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:47.528805 28039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:47.531646 28039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "a78cba9c7a384e2fa1ed793bee2933ea"
format_stamp: "Formatted at 2026-09-26 07:33:47 on dist-test-slave-qlq8"
server_key: "5d23e46015024b52ad25e8219ac85f86"
server_key_iv: "fe03123b1874ef95d77d1fa3c5610aa3"
server_key_version: "encryptionkey@0"
I20260926 07:33:47.532416 28039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "a78cba9c7a384e2fa1ed793bee2933ea"
format_stamp: "Formatted at 2026-09-26 07:33:47 on dist-test-slave-qlq8"
server_key: "5d23e46015024b52ad25e8219ac85f86"
server_key_iv: "fe03123b1874ef95d77d1fa3c5610aa3"
server_key_version: "encryptionkey@0"
I20260926 07:33:47.537405 28039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:33:47.540686 28053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:47.542057 28039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:33:47.542225 28039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "a78cba9c7a384e2fa1ed793bee2933ea"
format_stamp: "Formatted at 2026-09-26 07:33:47 on dist-test-slave-qlq8"
server_key: "5d23e46015024b52ad25e8219ac85f86"
server_key_iv: "fe03123b1874ef95d77d1fa3c5610aa3"
server_key_version: "encryptionkey@0"
I20260926 07:33:47.542433 28039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:47.548154 28039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:47.548885 28039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:47.549185 28039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:47.567044 28039 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:42969
I20260926 07:33:47.567075 28104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:42969 every 8 connection(s)
I20260926 07:33:47.568383 28039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:47.572001 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:47.575199 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28039
I20260926 07:33:47.575420 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:47.576254 23413 external_mini_cluster.cc:1428] Setting key 7709ce4a3f286178870fc20bb0e275ac
I20260926 07:33:47.584244 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.578255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.22.221.126:42969)
0926 07:33:47.579159 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:47.579168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:47.579207 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:33:47.580798 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:47.580818 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:47.581237 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:47.581630 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:47.582688 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:47.582693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:47.582704 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:47.582717 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:47.582767 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:47.582780 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:47.583003 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:47.583143 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:33:47.583413 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":312,"threads_started":1}
I20260926 07:33:47.584242 28105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea: Bootstrap starting.
I20260926 07:33:47.587350 28105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:47.588307 28105 log.cc:824] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:47.590698 28105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea: No bootstrap required, opened a new log
I20260926 07:33:47.598093 28105 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42969 } }
I20260926 07:33:47.598491 28105 raft_consensus.cc:374] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:47.598614 28105 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78cba9c7a384e2fa1ed793bee2933ea, State: Initialized, Role: FOLLOWER
I20260926 07:33:47.599153 28105 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42969 } }
I20260926 07:33:47.599428 28105 raft_consensus.cc:388] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:47.599560 28105 raft_consensus.cc:482] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:47.599730 28105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:47.601727 28105 raft_consensus.cc:504] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42969 } }
I20260926 07:33:47.602196 28105 leader_election.cc:302] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78cba9c7a384e2fa1ed793bee2933ea; no voters:
I20260926 07:33:47.602742 28105 leader_election.cc:288] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:47.602840 28110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:47.603652 28110 raft_consensus.cc:686] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [term 1 LEADER]: Becoming Leader. State: Replica: a78cba9c7a384e2fa1ed793bee2933ea, State: Running, Role: LEADER
I20260926 07:33:47.604198 28110 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42969 } }
I20260926 07:33:47.604827 28105 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:47.606814 28112 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78cba9c7a384e2fa1ed793bee2933ea. Latest consensus state: current_term: 1 leader_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42969 } } }
I20260926 07:33:47.607180 28112 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:47.607930 28111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78cba9c7a384e2fa1ed793bee2933ea" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42969 } } }
I20260926 07:33:47.608376 28111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:47.608682 28116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:47.613396 28116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:47.622072 28116 catalog_manager.cc:1347] Generated new cluster ID: b777b53256934f0993465f2074dd738c
I20260926 07:33:47.622179 28116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:47.641614 28116 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:47.642045 28116 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:47.656762 28116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea: Generated new TSK 0
I20260926 07:33:47.657734 28116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:33:47.668834 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42969
--builtin_ntp_servers=127.22.221.84:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:33:47.983987 28129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:47.984247 28129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:47.984336 28129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:47.991308 28129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:47.991400 28129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:47.991505 28129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:48.000227 28129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:48.001372 28129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:48.003023 28129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:48.016999 28137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:48.016947 28135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:48.017098 28134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:48.018726 28129 server_base.cc:1034] running on GCE node
I20260926 07:33:48.019373 28129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:48.020237 28129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:48.021425 28129 hybrid_clock.cc:648] HybridClock initialized: now 1790408028021366 us; error 51 us; skew 500 ppm
I20260926 07:33:48.021733 28129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:48.023978 28129 webserver.cc:460] Webserver started at http://127.22.221.65:43685/ using document root <none> and password file <none>
I20260926 07:33:48.024659 28129 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:48.024772 28129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:48.025101 28129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:48.028074 28129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "162a7d81cff248039019b5999359345e"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "14e2c008d0c96b52d9a047e9fd14d953"
server_key_iv: "0851fbbf1f00c5df05f836114f8b3267"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.028862 28129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "162a7d81cff248039019b5999359345e"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "14e2c008d0c96b52d9a047e9fd14d953"
server_key_iv: "0851fbbf1f00c5df05f836114f8b3267"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.033888 28129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:33:48.037289 28143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.038460 28129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:33:48.038640 28129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "162a7d81cff248039019b5999359345e"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "14e2c008d0c96b52d9a047e9fd14d953"
server_key_iv: "0851fbbf1f00c5df05f836114f8b3267"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.038859 28129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:48.044313 28129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:48.045073 28129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:48.045346 28129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:48.046340 28129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:48.047847 28129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:48.047968 28129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.048096 28129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:48.048195 28129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.081243 28129 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:40517
I20260926 07:33:48.081298 28255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:40517 every 8 connection(s)
I20260926 07:33:48.082566 28129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:48.086967 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28129
I20260926 07:33:48.087157 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:48.087841 23413 external_mini_cluster.cc:1428] Setting key 3ec8ea22fae34178f38a6dc3d73ef379
I20260926 07:33:48.090705 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42969
--builtin_ntp_servers=127.22.221.84:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:33:48.091384 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.085530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:52797 (local address 127.22.221.126:42969)
0926 07:33:48.085954 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.085958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.087259 (+ 1301us) server_negotiation.cc:407] Connection header received
0926 07:33:48.089069 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.089074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.089136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.089266 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.090611 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.090618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.090627 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.090642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.090688 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.090700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.090821 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.091016 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.091081 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":214,"threads_started":1}
I20260926 07:33:48.092301 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.086130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42969 (local address 127.22.221.65:52797)
0926 07:33:48.087014 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.087084 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:48.088803 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:48.089511 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:48.089526 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:48.089618 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:48.090325 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:48.090327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:48.090364 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:48.090444 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:48.090798 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:48.090808 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:48.090816 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:33:48.091094 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 07:33:48.091378 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260926 07:33:48.093813 28256 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42969
I20260926 07:33:48.094103 28256 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:48.094641 28256 heartbeater.cc:511] Master 127.22.221.126:42969 requested a full tablet report, sending...
I20260926 07:33:48.096632 28070 ts_manager.cc:194] Registered new tserver with Master: 162a7d81cff248039019b5999359345e (127.22.221.65:40517)
W20260926 07:33:48.408133 28260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:48.408377 28260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:48.408460 28260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:48.415436 28260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:48.415522 28260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:48.415625 28260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:48.424611 28260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:48.425822 28260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:48.427603 28260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:48.440230 28265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:48.440256 28268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:48.440704 28266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:48.441031 28260 server_base.cc:1034] running on GCE node
I20260926 07:33:48.441565 28260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:48.442466 28260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:48.443681 28260 hybrid_clock.cc:648] HybridClock initialized: now 1790408028443641 us; error 33 us; skew 500 ppm
I20260926 07:33:48.443984 28260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:48.446125 28260 webserver.cc:460] Webserver started at http://127.22.221.66:33265/ using document root <none> and password file <none>
I20260926 07:33:48.446841 28260 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:48.446966 28260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:48.447343 28260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:48.449967 28260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "7fca54aa75956839609169d70e207dfc"
server_key_iv: "38cfa038e3fa83d5009725c21dbc240b"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.450739 28260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "7fca54aa75956839609169d70e207dfc"
server_key_iv: "38cfa038e3fa83d5009725c21dbc240b"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.455603 28260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 07:33:48.458869 28274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.460561 28260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:48.460696 28260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "7fca54aa75956839609169d70e207dfc"
server_key_iv: "38cfa038e3fa83d5009725c21dbc240b"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.460870 28260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:48.469420 28260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:48.470395 28260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:48.470718 28260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:48.471762 28260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:48.473122 28260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:48.473244 28260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.473373 28260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:48.473471 28260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.504480 28260 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:40529
I20260926 07:33:48.504522 28386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:40529 every 8 connection(s)
I20260926 07:33:48.505869 28260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:48.508930 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28260
I20260926 07:33:48.509083 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:48.509716 23413 external_mini_cluster.cc:1428] Setting key 55e07e805fbf42134abb43fd240a57d6
I20260926 07:33:48.512867 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42969
--builtin_ntp_servers=127.22.221.84:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:33:48.514578 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.508393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:43975 (local address 127.22.221.126:42969)
0926 07:33:48.508574 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.508598 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.510261 (+ 1663us) server_negotiation.cc:407] Connection header received
0926 07:33:48.512292 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.512296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.512345 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.512426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.513889 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.513893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.513896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.513910 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.513941 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.513952 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.514051 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.514306 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.514354 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:33:48.515800 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.508921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42969 (local address 127.22.221.66:43975)
0926 07:33:48.510068 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.510121 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:48.512005 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:48.512691 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:48.512703 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:48.512824 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:48.513545 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:48.513550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:48.513602 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:48.513719 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:48.514068 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:48.514080 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:48.514092 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:33:48.514378 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 07:33:48.514622 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":337,"threads_started":1}
I20260926 07:33:48.517110 28387 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42969
I20260926 07:33:48.517417 28387 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:48.518026 28387 heartbeater.cc:511] Master 127.22.221.126:42969 requested a full tablet report, sending...
I20260926 07:33:48.519467 28070 ts_manager.cc:194] Registered new tserver with Master: ed4ce79fbf6b47aeb3fc7ca3f53affba (127.22.221.66:40529)
W20260926 07:33:48.826357 28390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:48.826609 28390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:48.826754 28390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:48.834038 28390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:48.834127 28390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:48.834304 28390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:48.843350 28390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:48.844574 28390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:48.846374 28390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:48.858891 28395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:48.858855 28396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:48.858944 28398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:48.860101 28390 server_base.cc:1034] running on GCE node
I20260926 07:33:48.860610 28390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:48.861436 28390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:48.862658 28390 hybrid_clock.cc:648] HybridClock initialized: now 1790408028862608 us; error 63 us; skew 500 ppm
I20260926 07:33:48.862960 28390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:48.865077 28390 webserver.cc:460] Webserver started at http://127.22.221.67:35429/ using document root <none> and password file <none>
I20260926 07:33:48.865758 28390 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:48.865890 28390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:48.866252 28390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:48.868953 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "8a1d900450c34ca7a62f51795ba2505e"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "984eb36282f1ff3a2148b2e8db689459"
server_key_iv: "6dc2c9b08193f786dcb86258c90309e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.869731 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "8a1d900450c34ca7a62f51795ba2505e"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "984eb36282f1ff3a2148b2e8db689459"
server_key_iv: "6dc2c9b08193f786dcb86258c90309e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.874475 28390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 07:33:48.877688 28404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.878866 28390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:48.879035 28390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "8a1d900450c34ca7a62f51795ba2505e"
format_stamp: "Formatted at 2026-09-26 07:33:48 on dist-test-slave-qlq8"
server_key: "984eb36282f1ff3a2148b2e8db689459"
server_key_iv: "6dc2c9b08193f786dcb86258c90309e9"
server_key_version: "encryptionkey@0"
I20260926 07:33:48.879271 28390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:48.884599 28390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:48.885346 28390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:48.885610 28390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:48.886471 28390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:48.887883 28390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:48.888008 28390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.888149 28390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:48.888240 28390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:48.920531 28390 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:46403
I20260926 07:33:48.920583 28516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:46403 every 8 connection(s)
I20260926 07:33:48.921825 28390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:48.922133 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28390
I20260926 07:33:48.922305 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:48.922884 23413 external_mini_cluster.cc:1428] Setting key b2649948a8dbd5100b6298c2f142be73
I20260926 07:33:48.930230 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.924562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:53753 (local address 127.22.221.126:42969)
0926 07:33:48.924751 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.924755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.926484 (+ 1729us) server_negotiation.cc:407] Connection header received
0926 07:33:48.928172 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.928176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.928226 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.928306 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.929667 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.929670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.929673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.929688 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.929717 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.929728 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.929820 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.929985 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.930031 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:33:48.931113 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.925279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42969 (local address 127.22.221.67:53753)
0926 07:33:48.926340 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.926381 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:48.927943 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:48.928502 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:48.928510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:48.928596 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:48.929347 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:48.929350 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:48.929387 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:48.929514 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:48.929811 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:48.929819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:48.929824 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:33:48.930023 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:33:48.930206 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260926 07:33:48.932396 28517 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42969
I20260926 07:33:48.932677 28517 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:48.933229 28517 heartbeater.cc:511] Master 127.22.221.126:42969 requested a full tablet report, sending...
I20260926 07:33:48.934535 28070 ts_manager.cc:194] Registered new tserver with Master: 8a1d900450c34ca7a62f51795ba2505e (127.22.221.67:46403)
I20260926 07:33:48.938117 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:48.944975 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.942406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.22.221.126:42969)
0926 07:33:48.942587 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.942609 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.943127 (+ 518us) server_negotiation.cc:407] Connection header received
0926 07:33:48.943326 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.943349 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.943409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.943507 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.944341 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.944347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.944351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.944368 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.944406 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.944417 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.944540 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.944656 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.944714 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:33:48.948661 28070 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:48.956082 28070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:48.977175 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.968632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.22.221.66:40529)
0926 07:33:48.969232 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.969245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.973646 (+ 4401us) server_negotiation.cc:407] Connection header received
0926 07:33:48.973894 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.973929 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.974163 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.974293 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.976283 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.976291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.976305 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.976338 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.976382 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.976397 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.976567 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.976685 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.976783 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":283,"threads_started":1}
I20260926 07:33:48.977505 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.973197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.22.221.65:40517)
0926 07:33:48.973806 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.973815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.974424 (+ 609us) server_negotiation.cc:407] Connection header received
0926 07:33:48.974664 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.974675 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.974873 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.974996 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.976545 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.976555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.976570 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.976600 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.976660 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.976681 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.976782 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.977056 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.977154 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":304,"threads_started":1}
I20260926 07:33:48.977623 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.967849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:46403 (local address 127.0.0.1:44884)
0926 07:33:48.972419 (+ 4570us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.972453 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:48.972564 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:48.973900 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:48.973911 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:48.974049 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:48.975004 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:48.975007 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:48.975035 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:48.975118 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:48.976998 (+ 1880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:48.977002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:48.977006 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:33:48.977134 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 07:33:48.977183 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4332,"thread_start_us":5402,"threads_started":1}
I20260926 07:33:48.977247 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.968045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:40529 (local address 127.0.0.1:58368)
0926 07:33:48.973526 (+ 5481us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.973547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:48.973735 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:48.975422 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:48.975428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:48.975455 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:48.976082 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:48.976084 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:48.976096 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:48.976127 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:48.976495 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:48.976505 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:48.976512 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:33:48.976686 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 07:33:48.976775 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5336,"thread_start_us":3973,"threads_started":1}
I20260926 07:33:48.977939 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.968106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.22.221.67:46403)
0926 07:33:48.973285 (+ 5179us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:48.973293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:48.973329 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:33:48.973413 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:48.973421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:48.973665 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:48.975300 (+ 1635us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:48.976760 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:48.976767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:48.976777 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:48.976794 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:48.976846 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:48.976867 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:48.977406 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:48.977473 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:33:48.977562 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5012,"thread_start_us":4731,"threads_started":1}
I20260926 07:33:48.978242 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.973777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53868)
0926 07:33:48.974325 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.974343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:48.974509 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:48.975544 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:48.975549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:48.975578 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:48.976280 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:48.976285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:48.976304 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:48.976356 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:48.976879 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:48.976885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:48.976891 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:33:48.977805 (+ 914us) client_negotiation.cc:240] Negotiation successful
0926 07:33:48.977858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
I20260926 07:33:48.981205 28452 tablet_service.cc:1467] Processing CreateTablet for tablet f8083ad529744727883a0e074bae5f59 (DEFAULT_TABLE table=test-table [id=bf3a683928e2473aab478f46750ceb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:48.981815 28191 tablet_service.cc:1467] Processing CreateTablet for tablet f8083ad529744727883a0e074bae5f59 (DEFAULT_TABLE table=test-table [id=bf3a683928e2473aab478f46750ceb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:48.981705 28322 tablet_service.cc:1467] Processing CreateTablet for tablet f8083ad529744727883a0e074bae5f59 (DEFAULT_TABLE table=test-table [id=bf3a683928e2473aab478f46750ceb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:48.982502 28452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8083ad529744727883a0e074bae5f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:48.983001 28191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8083ad529744727883a0e074bae5f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:48.983011 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8083ad529744727883a0e074bae5f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:48.992810 28535 tablet_bootstrap.cc:492] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Bootstrap starting.
I20260926 07:33:48.995879 28535 tablet_bootstrap.cc:654] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:48.997107 28535 log.cc:824] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:48.998121 28536 tablet_bootstrap.cc:492] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Bootstrap starting.
I20260926 07:33:48.999742 28538 tablet_bootstrap.cc:492] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Bootstrap starting.
I20260926 07:33:49.001227 28535 tablet_bootstrap.cc:492] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: No bootstrap required, opened a new log
I20260926 07:33:49.001508 28535 ts_tablet_manager.cc:1397] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 07:33:49.002055 28536 tablet_bootstrap.cc:654] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:49.003276 28536 log.cc:824] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:49.003436 28538 tablet_bootstrap.cc:654] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:49.004467 28538 log.cc:824] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:49.006676 28536 tablet_bootstrap.cc:492] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: No bootstrap required, opened a new log
I20260926 07:33:49.006978 28536 ts_tablet_manager.cc:1397] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 07:33:49.010243 28535 raft_consensus.cc:348] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.010596 28535 raft_consensus.cc:374] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:49.010702 28535 raft_consensus.cc:729] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1d900450c34ca7a62f51795ba2505e, State: Initialized, Role: FOLLOWER
I20260926 07:33:49.011392 28535 consensus_queue.cc:260] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.012470 28517 heartbeater.cc:503] Master 127.22.221.126:42969 was elected leader, sending a full tablet report...
I20260926 07:33:49.013345 28535 ts_tablet_manager.cc:1428] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260926 07:33:49.013629 28538 tablet_bootstrap.cc:492] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: No bootstrap required, opened a new log
I20260926 07:33:49.013917 28538 ts_tablet_manager.cc:1397] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260926 07:33:49.018126 28536 raft_consensus.cc:348] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.018569 28536 raft_consensus.cc:374] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:49.018682 28536 raft_consensus.cc:729] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ce79fbf6b47aeb3fc7ca3f53affba, State: Initialized, Role: FOLLOWER
I20260926 07:33:49.019203 28536 consensus_queue.cc:260] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.020255 28387 heartbeater.cc:503] Master 127.22.221.126:42969 was elected leader, sending a full tablet report...
I20260926 07:33:49.020718 28536 ts_tablet_manager.cc:1428] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 07:33:49.022537 28538 raft_consensus.cc:348] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.022965 28538 raft_consensus.cc:374] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:49.023067 28538 raft_consensus.cc:729] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162a7d81cff248039019b5999359345e, State: Initialized, Role: FOLLOWER
I20260926 07:33:49.023687 28538 consensus_queue.cc:260] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.024660 28256 heartbeater.cc:503] Master 127.22.221.126:42969 was elected leader, sending a full tablet report...
I20260926 07:33:49.025161 28538 ts_tablet_manager.cc:1428] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 07:33:49.058614 28542 raft_consensus.cc:482] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:49.058941 28542 raft_consensus.cc:504] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.060590 28542 leader_election.cc:288] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a1d900450c34ca7a62f51795ba2505e (127.22.221.67:46403), 162a7d81cff248039019b5999359345e (127.22.221.65:40517)
I20260926 07:33:49.063722 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.061097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:51577 (local address 127.22.221.65:40517)
0926 07:33:49.061295 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:49.061300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:49.061649 (+ 349us) server_negotiation.cc:407] Connection header received
0926 07:33:49.061857 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:49.061862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:49.061929 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:49.062023 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:49.063075 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:49.063080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:49.063085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:49.063104 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:49.063139 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:49.063154 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:49.063317 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:49.063395 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:33:49.063452 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:33:49.063719 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.061090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:58987 (local address 127.22.221.67:46403)
0926 07:33:49.061291 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:49.061296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:49.061903 (+ 607us) server_negotiation.cc:407] Connection header received
0926 07:33:49.061990 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:49.061998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:49.062057 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:49.062169 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:49.063075 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:49.063080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:49.063085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:49.063104 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:49.063150 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:49.063164 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:49.063316 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:49.063395 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 07:33:49.063452 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:33:49.063992 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.060832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.22.221.66:51577)
0926 07:33:49.061527 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.061544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:49.061708 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:49.062039 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:49.062042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:49.062059 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:49.062821 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:49.062826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:49.062848 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:49.062906 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:49.063284 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:49.063289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:49.063295 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:33:49.063436 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 07:33:49.063475 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":1,"thread_start_us":351,"threads_started":1}
I20260926 07:33:49.064030 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.060832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:46403 (local address 127.22.221.66:58987)
0926 07:33:49.061793 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.061811 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:49.061901 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:49.062225 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:49.062232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:49.062258 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:49.062897 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:33:49.062900 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:49.062911 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:33:49.062941 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:49.063284 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:49.063289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:49.063295 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:33:49.063408 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:33:49.063467 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":193,"thread_start_us":441,"threads_started":1}
I20260926 07:33:49.065083 28472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8083ad529744727883a0e074bae5f59" candidate_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1d900450c34ca7a62f51795ba2505e" is_pre_election: true
I20260926 07:33:49.065104 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8083ad529744727883a0e074bae5f59" candidate_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162a7d81cff248039019b5999359345e" is_pre_election: true
I20260926 07:33:49.065616 28472 raft_consensus.cc:2454] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4ce79fbf6b47aeb3fc7ca3f53affba in term 0.
I20260926 07:33:49.065649 28211 raft_consensus.cc:2454] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4ce79fbf6b47aeb3fc7ca3f53affba in term 0.
I20260926 07:33:49.066284 28278 leader_election.cc:302] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1d900450c34ca7a62f51795ba2505e, ed4ce79fbf6b47aeb3fc7ca3f53affba; no voters:
I20260926 07:33:49.066674 28542 raft_consensus.cc:2781] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:49.066816 28542 raft_consensus.cc:482] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:49.066917 28542 raft_consensus.cc:3037] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:49.069176 28542 raft_consensus.cc:504] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.069728 28542 leader_election.cc:288] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [CANDIDATE]: Term 1 election: Requested vote from peers 8a1d900450c34ca7a62f51795ba2505e (127.22.221.67:46403), 162a7d81cff248039019b5999359345e (127.22.221.65:40517)
I20260926 07:33:49.070124 28472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8083ad529744727883a0e074bae5f59" candidate_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1d900450c34ca7a62f51795ba2505e"
I20260926 07:33:49.070161 28211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8083ad529744727883a0e074bae5f59" candidate_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162a7d81cff248039019b5999359345e"
I20260926 07:33:49.070344 28472 raft_consensus.cc:3037] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:49.070400 28211 raft_consensus.cc:3037] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:49.072597 28472 raft_consensus.cc:2454] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4ce79fbf6b47aeb3fc7ca3f53affba in term 1.
I20260926 07:33:49.072734 28211 raft_consensus.cc:2454] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4ce79fbf6b47aeb3fc7ca3f53affba in term 1.
I20260926 07:33:49.073056 28278 leader_election.cc:302] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1d900450c34ca7a62f51795ba2505e, ed4ce79fbf6b47aeb3fc7ca3f53affba; no voters:
I20260926 07:33:49.073339 28542 raft_consensus.cc:2781] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:49.073796 28542 raft_consensus.cc:686] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 1 LEADER]: Becoming Leader. State: Replica: ed4ce79fbf6b47aeb3fc7ca3f53affba, State: Running, Role: LEADER
I20260926 07:33:49.074235 28542 consensus_queue.cc:237] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } }
I20260926 07:33:49.078881 28070 catalog_manager.cc:5495] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4ce79fbf6b47aeb3fc7ca3f53affba (127.22.221.66). New cstate: current_term: 1 leader_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4ce79fbf6b47aeb3fc7ca3f53affba" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 } health_report { overall_health: UNKNOWN } } }
W20260926 07:33:49.084925 28257 tablet.cc:2369] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:49.098338 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.096426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.22.221.66:40529)
0926 07:33:49.096606 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:49.096611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:49.096628 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:33:49.096778 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:49.096782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:49.096852 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:49.096933 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:49.097791 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:49.097795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:49.097798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:49.097811 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:49.097841 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:49.097852 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:49.097951 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:49.098016 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:33:49.098059 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:33:49.106297 28211 raft_consensus.cc:1261] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 1 FOLLOWER]: Refusing update from remote peer ed4ce79fbf6b47aeb3fc7ca3f53affba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:49.106314 28472 raft_consensus.cc:1261] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 1 FOLLOWER]: Refusing update from remote peer ed4ce79fbf6b47aeb3fc7ca3f53affba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:49.107442 28546 consensus_queue.cc:1035] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1d900450c34ca7a62f51795ba2505e" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:49.107808 28542 consensus_queue.cc:1035] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [LEADER]: Connected to new peer: Peer: permanent_uuid: "162a7d81cff248039019b5999359345e" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:49.123395 28549 mvcc.cc:205] Tried to move back new op lower bound from 7333511287208730624 to 7333511287091662848. Current Snapshot: MvccSnapshot[applied={T|T < 7333511287208730624 or (T in {7333511287208730624})}]
I20260926 07:33:49.127058 28550 mvcc.cc:205] Tried to move back new op lower bound from 7333511287208730624 to 7333511287091662848. Current Snapshot: MvccSnapshot[applied={T|T < 7333511287208730624}]
I20260926 07:33:49.133764 28070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:49.134034 28070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:49.138350 28070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a78cba9c7a384e2fa1ed793bee2933ea: Sending DeleteTablet for 3 replicas of tablet f8083ad529744727883a0e074bae5f59
I20260926 07:33:49.139797 28452 tablet_service.cc:1514] Processing DeleteTablet for tablet f8083ad529744727883a0e074bae5f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:49 UTC) from {username='slave'} at 127.0.0.1:44884
I20260926 07:33:49.139889 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet f8083ad529744727883a0e074bae5f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:49 UTC) from {username='slave'} at 127.0.0.1:58368
I20260926 07:33:49.139971 28191 tablet_service.cc:1514] Processing DeleteTablet for tablet f8083ad529744727883a0e074bae5f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:49 UTC) from {username='slave'} at 127.0.0.1:53868
I20260926 07:33:49.140744 28559 tablet_replica.cc:330] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: stopping tablet replica
I20260926 07:33:49.141112 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet f8083ad529744727883a0e074bae5f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:49 UTC) from {username='slave'} at 127.0.0.1:53868
I20260926 07:33:49.141189 28559 raft_consensus.cc:2229] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:49.141497 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28129
I20260926 07:33:49.141855 28559 raft_consensus.cc:2258] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:49.144018 28559 ts_tablet_manager.cc:1905] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:49.151347 28560 tablet_replica.cc:330] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: stopping tablet replica
I20260926 07:33:49.151579 28561 tablet_replica.cc:330] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: stopping tablet replica
I20260926 07:33:49.151742 28560 raft_consensus.cc:2229] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:49.151937 28561 raft_consensus.cc:2229] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:49.152192 28560 raft_consensus.cc:2258] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:49.152583 28561 raft_consensus.cc:2258] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:49.152742 28559 ts_tablet_manager.cc:1918] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:49.153061 28559 log.cc:1196] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/f8083ad529744727883a0e074bae5f59
I20260926 07:33:49.153671 28559 ts_tablet_manager.cc:1939] T f8083ad529744727883a0e074bae5f59 P 8a1d900450c34ca7a62f51795ba2505e: Deleting consensus metadata
I20260926 07:33:49.154418 28561 ts_tablet_manager.cc:1905] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:49.154410 28560 ts_tablet_manager.cc:1905] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:49.155645 28057 catalog_manager.cc:4841] TS 8a1d900450c34ca7a62f51795ba2505e (127.22.221.67:46403): tablet f8083ad529744727883a0e074bae5f59 (table test-table [id=bf3a683928e2473aab478f46750ceb05]) successfully deleted
I20260926 07:33:49.160539 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.143566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.22.221.65:40517)
0926 07:33:49.143730 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:49.143738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:49.157881 (+ 14143us) server_negotiation.cc:407] Connection header received
0926 07:33:49.158288 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:49.158295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:49.158398 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:49.158475 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:49.159712 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:49.159717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:49.159722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:49.159741 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:49.159775 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:49.159790 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:49.159885 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:49.160137 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:33:49.160201 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:33:49.161235 28560 ts_tablet_manager.cc:1918] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:49.161252 28561 ts_tablet_manager.cc:1918] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:49.161393 28561 log.cc:1196] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/f8083ad529744727883a0e074bae5f59
I20260926 07:33:49.161398 28560 log.cc:1196] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/f8083ad529744727883a0e074bae5f59
I20260926 07:33:49.161458 28251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:49.162019 28561 ts_tablet_manager.cc:1939] T f8083ad529744727883a0e074bae5f59 P ed4ce79fbf6b47aeb3fc7ca3f53affba: Deleting consensus metadata
I20260926 07:33:49.164003 28056 catalog_manager.cc:4841] TS ed4ce79fbf6b47aeb3fc7ca3f53affba (127.22.221.66:40529): tablet f8083ad529744727883a0e074bae5f59 (table test-table [id=bf3a683928e2473aab478f46750ceb05]) successfully deleted
I20260926 07:33:49.254958 28560 ts_tablet_manager.cc:1939] T f8083ad529744727883a0e074bae5f59 P 162a7d81cff248039019b5999359345e: Deleting consensus metadata
I20260926 07:33:49.255570 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28129
I20260926 07:33:49.277878 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28260
W20260926 07:33:49.277828 28056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.221.65:40517 (error 108)
I20260926 07:33:49.279966 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.279444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53884)
0926 07:33:49.279647 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.279793 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 07:33:49.280334 28056 catalog_manager.cc:4584] TS 162a7d81cff248039019b5999359345e (127.22.221.65:40517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8083ad529744727883a0e074bae5f59: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
I20260926 07:33:49.281411 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.279138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.22.221.66:40529)
0926 07:33:49.279355 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:49.279360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:49.279379 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:33:49.279479 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:49.279484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:49.279547 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:49.279668 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:49.280743 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:49.280747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:49.280750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:49.280764 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:49.280794 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:49.280809 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:49.280913 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:49.281069 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:33:49.281119 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:33:49.282075 28382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:49.324848 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.324394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53898)
0926 07:33:49.324601 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.324682 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:33:49.325892 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.325563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53908)
0926 07:33:49.325708 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.325788 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:33:49.333940 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.333572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53912)
0926 07:33:49.333730 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.333820 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:33:49.334844 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.334553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53920)
0926 07:33:49.334681 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.334743 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:33:49.374528 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.374016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53932)
0926 07:33:49.374276 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.374373 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 07:33:49.375638 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.375203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53942)
0926 07:33:49.375458 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.375536 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 07:33:49.391029 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28260
I20260926 07:33:49.409857 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28390
I20260926 07:33:49.411537 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.410998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53952)
0926 07:33:49.411196 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.411336 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 07:33:49.412645 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.412260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53956)
0926 07:33:49.412431 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.412518 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 07:33:49.412896 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.410608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.22.221.67:46403)
0926 07:33:49.410825 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:49.410833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:49.410855 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:33:49.410982 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:49.410988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:49.411092 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:49.411202 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:49.412230 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:49.412236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:49.412240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:49.412258 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:49.412295 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:49.412310 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:49.412440 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:49.412542 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:33:49.412589 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:33:49.413520 28512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:49.476287 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.475724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53964)
0926 07:33:49.475968 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.476080 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 07:33:49.477406 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.477041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53978)
0926 07:33:49.477200 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.477282 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:33:49.502871 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28390
I20260926 07:33:49.514722 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.514253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53988)
0926 07:33:49.514457 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.514556 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:33:49.515717 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.515383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40517 (local address 127.0.0.1:53998)
0926 07:33:49.515530 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.515606 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:33:49.519095 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28039
I20260926 07:33:49.520200 28100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:49.624840 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28039
2026-09-26T07:33:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2508 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 07:33:49.664987 23413 mini_oidc.cc:152] Starting JWKS server
I20260926 07:33:49.667537 23413 webserver.cc:460] Webserver started at https://127.0.0.1:34965/ using document root <none> and password file <none>
I20260926 07:33:49.667853 23413 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:33:49.668462 23413 webserver.cc:460] Webserver started at http://127.0.0.1:46289/ using document root <none> and password file <none>
2026-09-26T07:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:49Z Disabled control of system clock
I20260926 07:33:49.693800 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39797
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:39797
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34965/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/testchainca.cert with env {}
W20260926 07:33:50.004311 28573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:50.004659 28573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:50.004834 28573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:50.012111 28573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:50.012259 28573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:50.012341 28573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:50.012459 28573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:50.012557 28573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:33:50.012656 28573 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:33:50.012737 28573 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34965/jwks.json
I20260926 07:33:50.022722 28573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:39797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34965/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:50.024117 28573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:50.025437 28573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:50.037608 28578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:50.037608 28581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:50.037652 28579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:50.043869 28573 server_base.cc:1034] running on GCE node
I20260926 07:33:50.044737 28573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:50.046149 28573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:50.047343 28573 hybrid_clock.cc:648] HybridClock initialized: now 1790408030047300 us; error 41 us; skew 500 ppm
I20260926 07:33:50.047703 28573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:50.049947 28573 webserver.cc:460] Webserver started at http://127.22.221.126:42941/ using document root <none> and password file <none>
I20260926 07:33:50.050632 28573 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:50.050765 28573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:50.051064 28573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:50.053988 28573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "b2f551713cf34b6daa8a677fa2762c66"
format_stamp: "Formatted at 2026-09-26 07:33:50 on dist-test-slave-qlq8"
server_key: "6635f1089dd60f5aa30a80b2c9d9b658"
server_key_iv: "9143a4f80ab7c241f4556b7536682a56"
server_key_version: "encryptionkey@0"
I20260926 07:33:50.054772 28573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "b2f551713cf34b6daa8a677fa2762c66"
format_stamp: "Formatted at 2026-09-26 07:33:50 on dist-test-slave-qlq8"
server_key: "6635f1089dd60f5aa30a80b2c9d9b658"
server_key_iv: "9143a4f80ab7c241f4556b7536682a56"
server_key_version: "encryptionkey@0"
I20260926 07:33:50.059739 28573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:33:50.062950 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:50.064229 28573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:50.064412 28573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "b2f551713cf34b6daa8a677fa2762c66"
format_stamp: "Formatted at 2026-09-26 07:33:50 on dist-test-slave-qlq8"
server_key: "6635f1089dd60f5aa30a80b2c9d9b658"
server_key_iv: "9143a4f80ab7c241f4556b7536682a56"
server_key_version: "encryptionkey@0"
I20260926 07:33:50.064638 28573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:50.091192 28573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:50.092062 28573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:50.092375 28573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:50.110541 28573 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:39797
I20260926 07:33:50.110575 28638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:39797 every 8 connection(s)
I20260926 07:33:50.111948 28573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:50.115752 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:50.120123 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28573
I20260926 07:33:50.120371 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:50.121186 23413 external_mini_cluster.cc:1428] Setting key 4c1fdb22b7fc25708920aa98e3f39c72
I20260926 07:33:50.127446 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66: Bootstrap starting.
I20260926 07:33:50.130452 28639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:50.131592 28639 log.cc:824] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:50.134387 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66: No bootstrap required, opened a new log
I20260926 07:33:50.135107 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.123193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.22.221.126:39797)
0926 07:33:50.124213 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:50.124224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:50.124261 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:33:50.126192 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:50.126213 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:50.126646 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:50.127009 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:50.127608 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.128870 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.130692 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.131484 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.131697 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:50.132878 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:50.132885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:50.132897 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:50.133410 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:50.133482 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:50.133507 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:50.133639 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:50.133855 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:33:50.134144 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":372,"threads_started":1}
I20260926 07:33:50.138670 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:33:50.142654 28639 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f551713cf34b6daa8a677fa2762c66" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39797 } }
I20260926 07:33:50.143011 28639 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:50.143117 28639 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f551713cf34b6daa8a677fa2762c66, State: Initialized, Role: FOLLOWER
I20260926 07:33:50.143643 28639 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f551713cf34b6daa8a677fa2762c66" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39797 } }
I20260926 07:33:50.143889 28639 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:50.144052 28639 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:50.144235 28639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:50.146276 28639 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f551713cf34b6daa8a677fa2762c66" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39797 } }
I20260926 07:33:50.146770 28639 leader_election.cc:302] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f551713cf34b6daa8a677fa2762c66; no voters:
I20260926 07:33:50.147383 28639 leader_election.cc:288] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:50.147509 28644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:50.148200 28644 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [term 1 LEADER]: Becoming Leader. State: Replica: b2f551713cf34b6daa8a677fa2762c66, State: Running, Role: LEADER
I20260926 07:33:50.148730 28644 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f551713cf34b6daa8a677fa2762c66" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39797 } }
I20260926 07:33:50.149528 28639 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:50.151401 28645 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f551713cf34b6daa8a677fa2762c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f551713cf34b6daa8a677fa2762c66" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39797 } } }
I20260926 07:33:50.151755 28645 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:50.154505 28646 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f551713cf34b6daa8a677fa2762c66. Latest consensus state: current_term: 1 leader_uuid: "b2f551713cf34b6daa8a677fa2762c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f551713cf34b6daa8a677fa2762c66" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39797 } } }
I20260926 07:33:50.154767 28650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:50.154767 28646 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:50.160951 28650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:50.172458 28650 catalog_manager.cc:1347] Generated new cluster ID: 23fffc69b4cd43b1806dea9278246d99
I20260926 07:33:50.172621 28650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:50.186415 28650 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:50.187783 28650 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:50.197822 28650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f551713cf34b6daa8a677fa2762c66: Generated new TSK 0
I20260926 07:33:50.198588 28650 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:33:50.233775 28669 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 07:33:50.233950 28669 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:33:50.235067 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.212727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.22.221.126:39797)
0926 07:33:50.212922 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:50.212928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:50.213763 (+ 835us) server_negotiation.cc:407] Connection header received
0926 07:33:50.213889 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:50.213895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:50.214011 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:50.214166 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:33:50.214802 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.216024 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.217474 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.218191 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.218316 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:50.218805 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:33:50.234394 (+ 15589us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:33:50.234743 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 07:33:50.235184 28642 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 07:33:50.236384 28668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:50.212621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39797 (local address 127.0.0.1:37238)
0926 07:33:50.213395 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:50.213414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:50.213562 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:50.214230 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:50.214234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:50.214245 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:33:50.214604 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:50.214613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.216175 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.216179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:50.217327 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:50.217338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.218365 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.218370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:50.218496 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:50.218553 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:33:50.234691 (+ 16138us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:33:50.234945 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.221.126:39797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":386,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.221.126:39797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 07:33:50.237748 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28573
I20260926 07:33:50.238859 28634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:50.333268 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28573
2026-09-26T07:33:50Z chronyd exiting
I20260926 07:33:50.382789 23413 test_util.cc:175] -----------------------------------------------
I20260926 07:33:50.382951 23413 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407989013554-23413-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 07:33:50.386639 23413 mini_oidc.cc:152] Starting JWKS server
I20260926 07:33:50.388515 23413 webserver.cc:460] Webserver started at https://127.0.0.1:41693/ using document root <none> and password file <none>
I20260926 07:33:50.388769 23413 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:33:50.389424 23413 webserver.cc:460] Webserver started at http://127.0.0.1:39197/ using document root <none> and password file <none>
2026-09-26T07:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:50Z Disabled control of system clock
I20260926 07:33:50.414887 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40885
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:40885
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41693/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/testchainca.cert with env {}
W20260926 07:33:50.726996 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:50.727299 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:50.727413 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:50.734526 28676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:50.734627 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:50.734664 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:50.734723 28676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:50.734766 28676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:33:50.734819 28676 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:33:50.734858 28676 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41693/jwks.json
I20260926 07:33:50.744405 28676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:40885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41693/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:50.745628 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:50.746843 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:50.758227 28681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:50.758265 28682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:50.758323 28684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:50.759200 28676 server_base.cc:1034] running on GCE node
I20260926 07:33:50.760017 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:50.761307 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:50.762531 28676 hybrid_clock.cc:648] HybridClock initialized: now 1790408030762502 us; error 46 us; skew 500 ppm
I20260926 07:33:50.762874 28676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:50.765165 28676 webserver.cc:460] Webserver started at http://127.22.221.126:41437/ using document root <none> and password file <none>
I20260926 07:33:50.765836 28676 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:50.765971 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:50.766304 28676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:50.769073 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "e0dba8d24ef845ad88c1b25a2372fd69"
format_stamp: "Formatted at 2026-09-26 07:33:50 on dist-test-slave-qlq8"
server_key: "4d4922504b80119a27051ad4067c8ee9"
server_key_iv: "70dd9256401d91744a90bc05d2e4a775"
server_key_version: "encryptionkey@0"
I20260926 07:33:50.769850 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "e0dba8d24ef845ad88c1b25a2372fd69"
format_stamp: "Formatted at 2026-09-26 07:33:50 on dist-test-slave-qlq8"
server_key: "4d4922504b80119a27051ad4067c8ee9"
server_key_iv: "70dd9256401d91744a90bc05d2e4a775"
server_key_version: "encryptionkey@0"
I20260926 07:33:50.774750 28676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:33:50.778065 28690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:50.779482 28676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:50.779698 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "e0dba8d24ef845ad88c1b25a2372fd69"
format_stamp: "Formatted at 2026-09-26 07:33:50 on dist-test-slave-qlq8"
server_key: "4d4922504b80119a27051ad4067c8ee9"
server_key_iv: "70dd9256401d91744a90bc05d2e4a775"
server_key_version: "encryptionkey@0"
I20260926 07:33:50.779912 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:50.809340 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:50.810228 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:50.810545 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:50.828955 28676 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:40885
I20260926 07:33:50.828975 28741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:40885 every 8 connection(s)
I20260926 07:33:50.830235 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:50.834016 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:50.840637 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28676
I20260926 07:33:50.840829 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:50.841449 23413 external_mini_cluster.cc:1428] Setting key 6763087a61aa3bb00d2f30fe2c56a4c3
I20260926 07:33:50.847141 28742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69: Bootstrap starting.
I20260926 07:33:50.850925 28742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:50.852062 28742 log.cc:824] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:50.855197 28742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69: No bootstrap required, opened a new log
I20260926 07:33:50.857031 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.843353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.22.221.126:40885)
0926 07:33:50.844490 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:50.844506 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:50.844564 (+ 58us) server_negotiation.cc:407] Connection header received
0926 07:33:50.846903 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:50.846929 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:50.847520 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:50.847949 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:50.848470 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.850458 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.851830 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.852882 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.853041 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:50.854279 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:50.854290 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:50.854307 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:50.854986 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:50.855085 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:50.855109 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:50.855276 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:50.855517 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:33:50.855842 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":353,"threads_started":1}
I20260926 07:33:50.860734 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:33:50.863981 28742 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40885 } }
I20260926 07:33:50.864315 28742 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:50.864430 28742 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dba8d24ef845ad88c1b25a2372fd69, State: Initialized, Role: FOLLOWER
I20260926 07:33:50.864957 28742 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40885 } }
I20260926 07:33:50.865187 28742 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:50.865332 28742 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:50.865499 28742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:50.867743 28742 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40885 } }
I20260926 07:33:50.868269 28742 leader_election.cc:302] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0dba8d24ef845ad88c1b25a2372fd69; no voters:
I20260926 07:33:50.868796 28742 leader_election.cc:288] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:50.868942 28747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:50.869681 28747 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [term 1 LEADER]: Becoming Leader. State: Replica: e0dba8d24ef845ad88c1b25a2372fd69, State: Running, Role: LEADER
I20260926 07:33:50.870249 28747 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40885 } }
I20260926 07:33:50.870965 28742 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:50.873476 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40885 } } }
I20260926 07:33:50.873531 28749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0dba8d24ef845ad88c1b25a2372fd69. Latest consensus state: current_term: 1 leader_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0dba8d24ef845ad88c1b25a2372fd69" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40885 } } }
I20260926 07:33:50.873847 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:50.873927 28749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:50.875391 28755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:50.881513 28755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:50.891821 28755 catalog_manager.cc:1347] Generated new cluster ID: 5c1b39a684a245148f01b3f3e0cbbab4
I20260926 07:33:50.891957 28755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:50.926522 28755 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:50.927871 28755 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:50.938409 28755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0dba8d24ef845ad88c1b25a2372fd69: Generated new TSK 0
I20260926 07:33:50.939476 28755 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:33:50.983692 28772 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 07:33:50.983816 28772 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:33:50.985006 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.964752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.22.221.126:40885)
0926 07:33:50.964948 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:50.964956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:50.965384 (+ 428us) server_negotiation.cc:407] Connection header received
0926 07:33:50.965605 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:50.965611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:50.965708 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:50.965871 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:33:50.966335 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.967512 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.968926 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.969652 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.969777 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:50.970233 (+ 456us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:33:50.984363 (+ 14130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:33:50.984723 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 07:33:50.985117 28771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:50.964593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40885 (local address 127.0.0.1:33134)
0926 07:33:50.965282 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:50.965303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:50.965431 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:50.965909 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:50.965914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:50.965927 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:33:50.966172 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:50.966181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.967682 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.967686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:50.968786 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:50.968798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:50.969818 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:50.969822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:50.969957 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:50.969989 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:33:50.984610 (+ 14621us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:33:50.984811 (+ 201us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.221.126:40885: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":390,"threads_started":1}
W20260926 07:33:50.985128 28745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33134: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 07:33:50.986600 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28676
I20260926 07:33:50.987768 28737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:51.085923 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28676
2026-09-26T07:33:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 07:33:51.141202 23413 mini_oidc.cc:152] Starting JWKS server
I20260926 07:33:51.143046 23413 webserver.cc:460] Webserver started at https://127.0.0.1:35065/ using document root <none> and password file <none>
I20260926 07:33:51.143337 23413 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:33:51.143973 23413 webserver.cc:460] Webserver started at http://127.0.0.1:43683/ using document root <none> and password file <none>
2026-09-26T07:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:51Z Disabled control of system clock
I20260926 07:33:51.169926 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43163
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43163
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35065/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:33:51.482496 28779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:51.482786 28779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:51.482873 28779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:51.490002 28779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:51.490106 28779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:51.490150 28779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:51.490202 28779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:51.490240 28779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:33:51.490301 28779 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:33:51.490342 28779 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35065/jwks.json
I20260926 07:33:51.499471 28779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35065/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:51.500687 28779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:51.502429 28779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:51.514858 28785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:51.514897 28787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:51.514796 28784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:51.515172 28779 server_base.cc:1034] running on GCE node
I20260926 07:33:51.516317 28779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:51.517678 28779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:51.518923 28779 hybrid_clock.cc:648] HybridClock initialized: now 1790408031518877 us; error 56 us; skew 500 ppm
I20260926 07:33:51.519361 28779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:51.527148 28779 webserver.cc:460] Webserver started at http://127.22.221.126:35649/ using document root <none> and password file <none>
I20260926 07:33:51.527961 28779 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:51.528110 28779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:51.528482 28779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:51.531646 28779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "b3e416eaedb44d1897bc4631479c114c"
format_stamp: "Formatted at 2026-09-26 07:33:51 on dist-test-slave-qlq8"
server_key: "4f4f7b657f7f44f63a2f1809c0da135f"
server_key_iv: "ee480a8922688170f7bee47c5df6cbdd"
server_key_version: "encryptionkey@0"
I20260926 07:33:51.532518 28779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "b3e416eaedb44d1897bc4631479c114c"
format_stamp: "Formatted at 2026-09-26 07:33:51 on dist-test-slave-qlq8"
server_key: "4f4f7b657f7f44f63a2f1809c0da135f"
server_key_iv: "ee480a8922688170f7bee47c5df6cbdd"
server_key_version: "encryptionkey@0"
I20260926 07:33:51.538029 28779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:51.541529 28793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:51.542907 28779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 07:33:51.543108 28779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "b3e416eaedb44d1897bc4631479c114c"
format_stamp: "Formatted at 2026-09-26 07:33:51 on dist-test-slave-qlq8"
server_key: "4f4f7b657f7f44f63a2f1809c0da135f"
server_key_iv: "ee480a8922688170f7bee47c5df6cbdd"
server_key_version: "encryptionkey@0"
I20260926 07:33:51.543401 28779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:51.554509 28779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:51.555311 28779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:51.555613 28779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:51.573540 28779 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:43163
I20260926 07:33:51.573550 28844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:43163 every 8 connection(s)
I20260926 07:33:51.574846 28779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:51.575706 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28779
I20260926 07:33:51.575944 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:51.576614 23413 external_mini_cluster.cc:1428] Setting key 6565514f55556edc10053223eaf03975
I20260926 07:33:51.579217 28845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:51.591774 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:51.578496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.22.221.126:43163)
0926 07:33:51.579600 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:51.579615 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:51.579662 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:33:51.581890 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:51.581917 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:51.582506 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:51.582931 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:51.583470 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:51.585443 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:51.586702 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:51.587721 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:51.587901 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:51.589003 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:51.589013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:51.589035 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:51.589747 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:51.589842 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:51.589868 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:51.589983 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:51.590174 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:33:51.590504 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260926 07:33:51.593227 28845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c: Bootstrap starting.
I20260926 07:33:51.595801 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:33:51.596694 28845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:51.597664 28845 log.cc:824] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:51.600699 28845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c: No bootstrap required, opened a new log
I20260926 07:33:51.608271 28845 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e416eaedb44d1897bc4631479c114c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43163 } }
I20260926 07:33:51.608574 28845 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:51.608644 28845 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e416eaedb44d1897bc4631479c114c, State: Initialized, Role: FOLLOWER
I20260926 07:33:51.609094 28845 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e416eaedb44d1897bc4631479c114c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43163 } }
I20260926 07:33:51.609287 28845 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:51.609378 28845 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:51.609488 28845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:51.611557 28845 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e416eaedb44d1897bc4631479c114c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43163 } }
I20260926 07:33:51.612016 28845 leader_election.cc:302] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e416eaedb44d1897bc4631479c114c; no voters:
I20260926 07:33:51.612721 28845 leader_election.cc:288] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:51.612864 28850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:51.613687 28850 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [term 1 LEADER]: Becoming Leader. State: Replica: b3e416eaedb44d1897bc4631479c114c, State: Running, Role: LEADER
I20260926 07:33:51.614547 28850 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e416eaedb44d1897bc4631479c114c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43163 } }
I20260926 07:33:51.614768 28845 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:51.617717 28851 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e416eaedb44d1897bc4631479c114c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e416eaedb44d1897bc4631479c114c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43163 } } }
I20260926 07:33:51.617753 28852 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e416eaedb44d1897bc4631479c114c. Latest consensus state: current_term: 1 leader_uuid: "b3e416eaedb44d1897bc4631479c114c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e416eaedb44d1897bc4631479c114c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43163 } } }
I20260926 07:33:51.618181 28852 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:51.618181 28851 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:51.623126 28863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:51.627995 28863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:51.636705 28863 catalog_manager.cc:1347] Generated new cluster ID: 835bb6585c834d1ba696fcc5061dee19
I20260926 07:33:51.636813 28863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:51.650792 28863 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:51.652019 28863 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:51.656766 28863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3e416eaedb44d1897bc4631479c114c: Generated new TSK 0
I20260926 07:33:51.657527 28863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:33:51.677215 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:51.668828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.22.221.126:43163)
0926 07:33:51.669024 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:51.669031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:51.669436 (+ 405us) server_negotiation.cc:407] Connection header received
0926 07:33:51.669649 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:51.669655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:51.669756 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:51.669911 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:33:51.670378 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:51.671508 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:51.672961 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:51.673635 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:51.673786 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:51.674237 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:33:51.676593 (+ 2356us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35065/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:33:51.676926 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35065/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:33:51.677341 28848 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35065/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 07:33:51.679445 28874 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:51.668684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43163 (local address 127.0.0.1:53480)
0926 07:33:51.669339 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:51.669354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:51.669466 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:51.669927 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:51.669930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:51.669940 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:33:51.670233 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:51.670241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:51.671731 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:51.671735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:51.672809 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:51.672820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:51.673820 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:51.673824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:51.673942 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:51.673972 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:33:51.676816 (+ 2844us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35065/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:33:51.678931 (+ 2115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.221.126:43163: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35065/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":1807,"thread_start_us":388,"threads_started":1}
I20260926 07:33:51.680440 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28779
I20260926 07:33:51.681565 28840 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 07:33:51.695510 28875 webserver.cc:533] Webserver: sslize failed:
E20260926 07:33:51.695657 28875 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:33:51.775418 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28779
2026-09-26T07:33:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (688 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T07:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:51Z Disabled control of system clock
I20260926 07:33:51.855957 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46073
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:46073
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/testchainca.cert with env {}
W20260926 07:33:52.170241 28880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:52.170524 28880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:52.170676 28880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:52.177889 28880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:52.177991 28880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:52.178062 28880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:52.178184 28880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:52.178251 28880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:33:52.178363 28880 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:33:52.178442 28880 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 07:33:52.188356 28880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:46073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:52.189675 28880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:52.190972 28880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:52.202778 28885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:52.202857 28886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:52.202883 28888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:52.203758 28880 server_base.cc:1034] running on GCE node
I20260926 07:33:52.204492 28880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:52.205767 28880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:52.206969 28880 hybrid_clock.cc:648] HybridClock initialized: now 1790408032206943 us; error 36 us; skew 500 ppm
I20260926 07:33:52.207343 28880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:33:52.209537 28880 webserver.cc:460] Webserver started at http://127.22.221.126:34659/ using document root <none> and password file <none>
I20260926 07:33:52.210161 28880 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:52.210247 28880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:52.210492 28880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:52.213268 28880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "fa01c09bd3f949d098ca49a42f64bd92"
format_stamp: "Formatted at 2026-09-26 07:33:52 on dist-test-slave-qlq8"
server_key: "b667429bf12eb23097708c663d03d7fd"
server_key_iv: "9c8345e150cde457adc49163883f9770"
server_key_version: "encryptionkey@0"
I20260926 07:33:52.213991 28880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "fa01c09bd3f949d098ca49a42f64bd92"
format_stamp: "Formatted at 2026-09-26 07:33:52 on dist-test-slave-qlq8"
server_key: "b667429bf12eb23097708c663d03d7fd"
server_key_iv: "9c8345e150cde457adc49163883f9770"
server_key_version: "encryptionkey@0"
I20260926 07:33:52.218757 28880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 07:33:52.222065 28894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:52.223582 28880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:52.223709 28880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "fa01c09bd3f949d098ca49a42f64bd92"
format_stamp: "Formatted at 2026-09-26 07:33:52 on dist-test-slave-qlq8"
server_key: "b667429bf12eb23097708c663d03d7fd"
server_key_iv: "9c8345e150cde457adc49163883f9770"
server_key_version: "encryptionkey@0"
I20260926 07:33:52.223874 28880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:52.247773 28880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:52.248521 28880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:52.248754 28880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:52.266587 28880 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:46073
I20260926 07:33:52.266611 28945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:46073 every 8 connection(s)
I20260926 07:33:52.267918 28880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:52.271775 28946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:52.273538 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 28880
I20260926 07:33:52.273733 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:52.274390 23413 external_mini_cluster.cc:1428] Setting key 9c4d68b1db04981abd5aa64c1729fdd7
I20260926 07:33:52.283795 28946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92: Bootstrap starting.
I20260926 07:33:52.287413 28946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:52.288264 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:52.276308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.22.221.126:46073)
0926 07:33:52.277308 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:52.277318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:52.277356 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:33:52.279341 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:52.279372 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:52.279911 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:52.280235 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:52.280791 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:52.282092 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:52.283860 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:52.284545 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:52.284673 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:52.285923 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:52.285930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:52.285941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:33:52.286504 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:52.286589 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:33:52.286608 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:52.286689 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:52.286915 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:33:52.287288 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":336,"threads_started":1}
I20260926 07:33:52.288465 28946 log.cc:824] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:52.291656 28946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92: No bootstrap required, opened a new log
I20260926 07:33:52.292372 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:33:52.299997 28946 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa01c09bd3f949d098ca49a42f64bd92" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46073 } }
I20260926 07:33:52.300343 28946 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:52.300447 28946 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa01c09bd3f949d098ca49a42f64bd92, State: Initialized, Role: FOLLOWER
I20260926 07:33:52.300987 28946 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa01c09bd3f949d098ca49a42f64bd92" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46073 } }
I20260926 07:33:52.301236 28946 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:52.301378 28946 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:52.301537 28946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:52.303679 28946 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa01c09bd3f949d098ca49a42f64bd92" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46073 } }
I20260926 07:33:52.304155 28946 leader_election.cc:302] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa01c09bd3f949d098ca49a42f64bd92; no voters:
I20260926 07:33:52.304706 28946 leader_election.cc:288] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:52.304834 28951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:52.305889 28951 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [term 1 LEADER]: Becoming Leader. State: Replica: fa01c09bd3f949d098ca49a42f64bd92, State: Running, Role: LEADER
I20260926 07:33:52.306406 28951 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa01c09bd3f949d098ca49a42f64bd92" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46073 } }
I20260926 07:33:52.306797 28946 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:52.309322 28952 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa01c09bd3f949d098ca49a42f64bd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa01c09bd3f949d098ca49a42f64bd92" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46073 } } }
I20260926 07:33:52.309675 28952 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:52.310096 28953 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa01c09bd3f949d098ca49a42f64bd92. Latest consensus state: current_term: 1 leader_uuid: "fa01c09bd3f949d098ca49a42f64bd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa01c09bd3f949d098ca49a42f64bd92" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46073 } } }
I20260926 07:33:52.311559 28953 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:52.312861 28958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:52.318323 28958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:52.327152 28958 catalog_manager.cc:1347] Generated new cluster ID: f03bceb7bbaf466fabfbd7f9938e2b40
I20260926 07:33:52.327287 28958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:52.371663 28958 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:52.372962 28958 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:52.381889 28958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa01c09bd3f949d098ca49a42f64bd92: Generated new TSK 0
I20260926 07:33:52.382679 28958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:33:52.444002 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:52.404027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.22.221.126:46073)
0926 07:33:52.404208 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:52.404214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:52.404554 (+ 340us) server_negotiation.cc:407] Connection header received
0926 07:33:52.404754 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:52.404759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:52.404851 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:52.405064 (+ 213us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:33:52.405474 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:52.406605 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:52.408403 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:52.409112 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:52.409233 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:52.409682 (+ 449us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:33:52.443288 (+ 33606us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:33:52.443673 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41538: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 07:33:52.444137 28949 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41538: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 07:33:52.444154 28975 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:52.403896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:46073 (local address 127.0.0.1:41538)
0926 07:33:52.404453 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:52.404469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:52.404581 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:52.405090 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:52.405093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:52.405103 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:33:52.405339 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:52.405346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:52.406769 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:52.406773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:52.408268 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:52.408280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:52.409280 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:52.409284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:52.409399 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:52.409441 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:33:52.443581 (+ 34140us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:33:52.443825 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.221.126:46073: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":302,"threads_started":1}
I20260926 07:33:52.446027 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 28880
I20260926 07:33:52.447127 28941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:52.549515 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 28880
2026-09-26T07:33:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (753 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (404 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (400 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:53 dist-test-slave-qlq8 krb5kdc[28990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:53 dist-test-slave-qlq8 krb5kdc[28990](info): set up 2 sockets
Sep 26 07:33:53 dist-test-slave-qlq8 krb5kdc[28990](info): commencing operation
krb5kdc: starting...
W20260926 07:33:54.444039 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:54 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:33:54.642406 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43901
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:54.952011 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:54.952279 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:54.952431 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:54.959471 29003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:54.959566 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:54.959705 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:54.959800 29003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:54.959877 29003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:54.969223 29003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:54.970443 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:54.971760 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:54.983266 29008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:54.983346 29009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:54.983374 29011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:54.984222 29003 server_base.cc:1034] running on GCE node
I20260926 07:33:54.984891 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:54.986075 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:54.987313 29003 hybrid_clock.cc:648] HybridClock initialized: now 1790408034987269 us; error 42 us; skew 500 ppm
I20260926 07:33:54.987635 29003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:54 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408034, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:54.990909 29003 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:33:54.992383 29003 webserver.cc:460] Webserver started at http://127.22.221.126:36473/ using document root <none> and password file <none>
I20260926 07:33:54.993049 29003 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:54.993192 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:54.993494 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:54.996387 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "eaa7df46e2344800b1ffbae1cd350611"
format_stamp: "Formatted at 2026-09-26 07:33:54 on dist-test-slave-qlq8"
server_key: "998b726427efb95256938ee011f12c53"
server_key_iv: "05d208def3b6059385e1a5ac82006d05"
server_key_version: "encryptionkey@0"
I20260926 07:33:54.997144 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "eaa7df46e2344800b1ffbae1cd350611"
format_stamp: "Formatted at 2026-09-26 07:33:54 on dist-test-slave-qlq8"
server_key: "998b726427efb95256938ee011f12c53"
server_key_iv: "05d208def3b6059385e1a5ac82006d05"
server_key_version: "encryptionkey@0"
I20260926 07:33:55.002050 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:33:55.005285 29018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:55.006495 29003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:55.006670 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "eaa7df46e2344800b1ffbae1cd350611"
format_stamp: "Formatted at 2026-09-26 07:33:54 on dist-test-slave-qlq8"
server_key: "998b726427efb95256938ee011f12c53"
server_key_iv: "05d208def3b6059385e1a5ac82006d05"
server_key_version: "encryptionkey@0"
I20260926 07:33:55.006857 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:55.018843 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:55.022359 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:55.022630 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:55.040508 29003 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:43901
I20260926 07:33:55.040539 29069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:43901 every 8 connection(s)
I20260926 07:33:55.041901 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:55.045689 29070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:55.049410 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29003
I20260926 07:33:55.049633 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:55.050239 23413 external_mini_cluster.cc:1428] Setting key b3a1584e0dc593787cb9a4ca3bdb0679
I20260926 07:33:55.057163 29070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611: Bootstrap starting.
I20260926 07:33:55.060717 29070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:55.061858 29070 log.cc:824] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:55.065238 29070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611: No bootstrap required, opened a new log
Sep 26 07:33:55 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:55.072896 29070 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa7df46e2344800b1ffbae1cd350611" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43901 } }
I20260926 07:33:55.073289 29070 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:55.073421 29070 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa7df46e2344800b1ffbae1cd350611, State: Initialized, Role: FOLLOWER
I20260926 07:33:55.073689 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:55.052144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.22.221.126:43901)
0926 07:33:55.052870 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:55.052893 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:55.052933 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:33:55.055312 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:55.055333 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:55.055697 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:55.055994 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:55.057815 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:55.059148 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:55.060973 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:55.061632 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:55.061769 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:55.066318 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:55.066325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:55.066338 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:55.066378 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:55.069580 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:55.070281 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:55.070291 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:55.070304 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:55.070470 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:55.070853 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:55.070858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:55.070861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:55.071439 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:55.071764 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:55.072178 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 07:33:55.072562 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":343,"threads_started":1}
I20260926 07:33:55.074081 29070 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa7df46e2344800b1ffbae1cd350611" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43901 } }
I20260926 07:33:55.074355 29070 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:55.074505 29070 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:55.074692 29070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:55.077525 29070 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa7df46e2344800b1ffbae1cd350611" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43901 } }
I20260926 07:33:55.078001 29070 leader_election.cc:302] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa7df46e2344800b1ffbae1cd350611; no voters:
I20260926 07:33:55.078538 29070 leader_election.cc:288] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:55.078677 29075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:55.079468 29075 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [term 1 LEADER]: Becoming Leader. State: Replica: eaa7df46e2344800b1ffbae1cd350611, State: Running, Role: LEADER
I20260926 07:33:55.079996 29075 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa7df46e2344800b1ffbae1cd350611" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43901 } }
I20260926 07:33:55.080628 29070 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:55.082530 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa7df46e2344800b1ffbae1cd350611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa7df46e2344800b1ffbae1cd350611" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43901 } } }
I20260926 07:33:55.082947 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:55.084174 29077 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa7df46e2344800b1ffbae1cd350611. Latest consensus state: current_term: 1 leader_uuid: "eaa7df46e2344800b1ffbae1cd350611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa7df46e2344800b1ffbae1cd350611" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43901 } } }
I20260926 07:33:55.084606 29077 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:55.092235 29082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:55.097049 29082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:55.105759 29082 catalog_manager.cc:1347] Generated new cluster ID: 8b15981e11da47c7b872baa5ac693dc9
I20260926 07:33:55.105891 29082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:55.127061 29082 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:55.128443 29082 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:55.137266 29082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa7df46e2344800b1ffbae1cd350611: Generated new TSK 0
I20260926 07:33:55.138072 29082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:33:55.184533 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43901
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:55.501194 29096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:55.501441 29096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:55.501528 29096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:55.508453 29096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:55.508553 29096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:55.508661 29096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:55.518230 29096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:55.519448 29096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:55.520721 29096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:55.533617 29101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:55.533658 29104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:55.533582 29102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:55.534628 29096 server_base.cc:1034] running on GCE node
I20260926 07:33:55.535210 29096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:55.536125 29096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:55.537313 29096 hybrid_clock.cc:648] HybridClock initialized: now 1790408035537257 us; error 51 us; skew 500 ppm
I20260926 07:33:55.537560 29096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:55 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408035, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:55.540635 29096 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:33:55.541993 29096 webserver.cc:460] Webserver started at http://127.22.221.65:39113/ using document root <none> and password file <none>
I20260926 07:33:55.542650 29096 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:55.542779 29096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:55.543093 29096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:55.546012 29096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "81eac576b1a04ac285c5e56912d3dace"
format_stamp: "Formatted at 2026-09-26 07:33:55 on dist-test-slave-qlq8"
server_key: "d08bb718f9d33c68d7ac9b1824ee990a"
server_key_iv: "4922534aef2bac445620aaee2f79899c"
server_key_version: "encryptionkey@0"
I20260926 07:33:55.546799 29096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "81eac576b1a04ac285c5e56912d3dace"
format_stamp: "Formatted at 2026-09-26 07:33:55 on dist-test-slave-qlq8"
server_key: "d08bb718f9d33c68d7ac9b1824ee990a"
server_key_iv: "4922534aef2bac445620aaee2f79899c"
server_key_version: "encryptionkey@0"
I20260926 07:33:55.551826 29096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 07:33:55.554955 29111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:55.556211 29096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:33:55.556382 29096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "81eac576b1a04ac285c5e56912d3dace"
format_stamp: "Formatted at 2026-09-26 07:33:55 on dist-test-slave-qlq8"
server_key: "d08bb718f9d33c68d7ac9b1824ee990a"
server_key_iv: "4922534aef2bac445620aaee2f79899c"
server_key_version: "encryptionkey@0"
I20260926 07:33:55.556572 29096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:55.573257 29096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:55.576958 29096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:55.577245 29096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:55.578152 29096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:55.579427 29096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:55.579497 29096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:55.579674 29096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:55.579734 29096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:55.611090 29096 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:40421
I20260926 07:33:55.611147 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:40421 every 8 connection(s)
I20260926 07:33:55.612460 29096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:55.622766 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29096
I20260926 07:33:55.622957 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:55.623850 23413 external_mini_cluster.cc:1428] Setting key faa19d32d3f91642fd86b1320ec4b320
Sep 26 07:33:55 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408035, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:55.632679 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:55.614983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:41957 (local address 127.22.221.126:43901)
0926 07:33:55.615471 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:55.615475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:55.616451 (+ 976us) server_negotiation.cc:407] Connection header received
0926 07:33:55.618309 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:55.618315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:55.618406 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:55.618534 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:55.620062 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:55.620820 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:55.622399 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:55.622816 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:55.622965 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:55.627056 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:55.627061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:55.627064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:55.627096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:55.629646 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:55.630348 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:55.630354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:55.630359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:55.630479 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:55.630908 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:55.630913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:55.630916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:55.631199 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:55.631406 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:55.632089 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 07:33:55.632322 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":288,"threads_started":1}
I20260926 07:33:55.633726 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:55.615478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43901 (local address 127.22.221.65:41957)
0926 07:33:55.616258 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:55.616305 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:55.618025 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:55.618766 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:55.618774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:55.619341 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:55.619876 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:55.619894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:55.620994 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:55.620998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:55.622251 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:55.622262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:55.623126 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:55.623136 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:55.623471 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:55.624332 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:55.624367 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:55.626834 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:55.629790 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:55.629798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:55.629814 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:55.630222 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:55.630588 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:55.630593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:55.630597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:55.630790 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:55.631384 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:55.631392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:55.631731 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 07:33:55.631991 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 07:33:55.632504 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":293,"threads_started":1}
I20260926 07:33:55.635365 29224 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43901
I20260926 07:33:55.635675 29224 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:55.636483 29224 heartbeater.cc:511] Master 127.22.221.126:43901 requested a full tablet report, sending...
I20260926 07:33:55.638578 29035 ts_manager.cc:194] Registered new tserver with Master: 81eac576b1a04ac285c5e56912d3dace (127.22.221.65:40421)
I20260926 07:33:55.640120 29035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:41957
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:33:55.670449 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43901
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:55.981834 29230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:55.982137 29230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:55.982309 29230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:55.989372 29230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:55.989501 29230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:55.989686 29230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:55.998976 29230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:56.000205 29230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:56.001482 29230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:56.014679 29236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:56.014794 29235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:56.014678 29238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:56.015407 29230 server_base.cc:1034] running on GCE node
I20260926 07:33:56.015987 29230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:56.017033 29230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:56.018289 29230 hybrid_clock.cc:648] HybridClock initialized: now 1790408036018232 us; error 47 us; skew 500 ppm
I20260926 07:33:56.018620 29230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.021642 29230 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:33:56.022971 29230 webserver.cc:460] Webserver started at http://127.22.221.66:38365/ using document root <none> and password file <none>
I20260926 07:33:56.023823 29230 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:56.023973 29230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:56.024287 29230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:56.027077 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "74a9d110dba0413fa8472b0552a0c343"
format_stamp: "Formatted at 2026-09-26 07:33:56 on dist-test-slave-qlq8"
server_key: "3abf0d13b011944072428c27d1004deb"
server_key_iv: "69a35df982058688e93f9d8252385744"
server_key_version: "encryptionkey@0"
I20260926 07:33:56.027890 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "74a9d110dba0413fa8472b0552a0c343"
format_stamp: "Formatted at 2026-09-26 07:33:56 on dist-test-slave-qlq8"
server_key: "3abf0d13b011944072428c27d1004deb"
server_key_iv: "69a35df982058688e93f9d8252385744"
server_key_version: "encryptionkey@0"
I20260926 07:33:56.032717 29230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:33:56.036083 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:56.037390 29230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:56.037573 29230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "74a9d110dba0413fa8472b0552a0c343"
format_stamp: "Formatted at 2026-09-26 07:33:56 on dist-test-slave-qlq8"
server_key: "3abf0d13b011944072428c27d1004deb"
server_key_iv: "69a35df982058688e93f9d8252385744"
server_key_version: "encryptionkey@0"
I20260926 07:33:56.037782 29230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:56.055675 29230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:56.059675 29230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:56.059959 29230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:56.060940 29230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:56.062239 29230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:56.062343 29230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:56.062487 29230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:56.062559 29230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:56.095719 29230 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:37467
I20260926 07:33:56.095774 29357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:37467 every 8 connection(s)
I20260926 07:33:56.097003 29230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:56.098443 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29230
I20260926 07:33:56.098641 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:56.099102 23413 external_mini_cluster.cc:1428] Setting key 109527399a3bbe6a5868a60dfb2a67c1
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:56.117660 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.099622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:48743 (local address 127.22.221.126:43901)
0926 07:33:56.099912 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.099918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.101212 (+ 1294us) server_negotiation.cc:407] Connection header received
0926 07:33:56.103103 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.103111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.103202 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.103330 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.104755 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.105956 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.107585 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.108241 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.108343 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.111927 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.111933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.111936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.111969 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.114596 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.115355 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.115363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.115369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.115517 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.115961 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.115970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.115972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.116232 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.116394 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.117074 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.117308 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260926 07:33:56.118664 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.100250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43901 (local address 127.22.221.66:48743)
0926 07:33:56.101058 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:56.101103 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:56.102819 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:56.103519 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:56.103527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:56.104023 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:56.104582 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:56.104598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.106121 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.106125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:56.107446 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:56.107458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.108430 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.108434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:56.108622 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.109314 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:56.109340 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:56.111702 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:56.114751 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:56.114760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:56.114778 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:56.115211 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:56.115633 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:56.115641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:56.115645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:56.115837 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:56.116366 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:56.116373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:56.116721 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 07:33:56.116986 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 07:33:56.117532 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:33:56.120193 29358 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43901
I20260926 07:33:56.120550 29358 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:56.121280 29358 heartbeater.cc:511] Master 127.22.221.126:43901 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
I20260926 07:33:56.122926 29035 ts_manager.cc:194] Registered new tserver with Master: 74a9d110dba0413fa8472b0552a0c343 (127.22.221.66:37467)
I20260926 07:33:56.124243 29035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:48743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:33:56.146478 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43901
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:56.459573 29363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:56.459899 29363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:56.460000 29363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:56.470963 29363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:56.471066 29363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:56.471180 29363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:33:56.480969 29363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:43901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:56.482254 29363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:56.483631 29363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:56.496803 29368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:56.496845 29371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:56.496879 29369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:56.497705 29363 server_base.cc:1034] running on GCE node
I20260926 07:33:56.498306 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:56.499274 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:56.500512 29363 hybrid_clock.cc:648] HybridClock initialized: now 1790408036500466 us; error 59 us; skew 500 ppm
I20260926 07:33:56.500833 29363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.504002 29363 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:33:56.505393 29363 webserver.cc:460] Webserver started at http://127.22.221.67:40079/ using document root <none> and password file <none>
I20260926 07:33:56.506089 29363 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:56.506242 29363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:56.506551 29363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:56.510002 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "1f91f0cd3c3d45188e8fad26d92e56fb"
format_stamp: "Formatted at 2026-09-26 07:33:56 on dist-test-slave-qlq8"
server_key: "14e5fb43471c13a483c3c43cecd07e02"
server_key_iv: "f3ee9736109ae4869dc0d78844725568"
server_key_version: "encryptionkey@0"
I20260926 07:33:56.510898 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "1f91f0cd3c3d45188e8fad26d92e56fb"
format_stamp: "Formatted at 2026-09-26 07:33:56 on dist-test-slave-qlq8"
server_key: "14e5fb43471c13a483c3c43cecd07e02"
server_key_iv: "f3ee9736109ae4869dc0d78844725568"
server_key_version: "encryptionkey@0"
I20260926 07:33:56.516129 29363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:33:56.519584 29378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:56.520906 29363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:33:56.521080 29363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "1f91f0cd3c3d45188e8fad26d92e56fb"
format_stamp: "Formatted at 2026-09-26 07:33:56 on dist-test-slave-qlq8"
server_key: "14e5fb43471c13a483c3c43cecd07e02"
server_key_iv: "f3ee9736109ae4869dc0d78844725568"
server_key_version: "encryptionkey@0"
I20260926 07:33:56.521296 29363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:56.546700 29363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:56.551023 29363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:56.551376 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:56.552318 29363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:56.553617 29363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:56.553715 29363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:56.553861 29363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:56.553921 29363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:56.585952 29363 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:45657
I20260926 07:33:56.586004 29490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:45657 every 8 connection(s)
I20260926 07:33:56.587296 29363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:56.594633 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29363
I20260926 07:33:56.594815 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:56.595325 23413 external_mini_cluster.cc:1428] Setting key 3ecfd1696d36398ea9e9ee16c6fa5428
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:33:56.606612 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.589860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:45587 (local address 127.22.221.126:43901)
0926 07:33:56.590036 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.590040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.591119 (+ 1079us) server_negotiation.cc:407] Connection header received
0926 07:33:56.592982 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.592987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.593081 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.593197 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.594919 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.595846 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.597874 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.598373 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.598469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.601610 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.601614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.601618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.601647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.604219 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.604821 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.604825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.604827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.604939 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.605294 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.605297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.605299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.605531 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.605643 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.606228 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.606376 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:33:56.607537 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.590253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43901 (local address 127.22.221.67:45587)
0926 07:33:56.590947 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:56.590982 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:56.592688 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:56.593427 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:56.593435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:56.593926 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:56.594687 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:56.594716 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.596018 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.596023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:56.597736 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:56.597748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.598556 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.598561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:56.598762 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.599385 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:56.599406 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:56.601411 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:56.604331 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:56.604337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:56.604347 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:56.604724 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:56.605029 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:56.605032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:56.605034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:56.605205 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:56.605634 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:56.605639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:56.605912 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 07:33:56.606144 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 07:33:56.606477 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":283,"threads_started":1}
I20260926 07:33:56.608917 29491 heartbeater.cc:347] Connected to a master server at 127.22.221.126:43901
I20260926 07:33:56.609261 29491 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:56.610141 29491 heartbeater.cc:511] Master 127.22.221.126:43901 requested a full tablet report, sending...
I20260926 07:33:56.611766 29035 ts_manager.cc:194] Registered new tserver with Master: 1f91f0cd3c3d45188e8fad26d92e56fb (127.22.221.67:45657)
I20260926 07:33:56.612844 29035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:45587
I20260926 07:33:56.622186 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:56.642849 29224 heartbeater.cc:503] Master 127.22.221.126:43901 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/test-admin.keytab.
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.648867 23413 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 07:33:56.653934 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:56.651407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.22.221.65:40421)
0926 07:33:56.652135 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.652142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.652168 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:33:56.652222 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.652228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.652436 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.652604 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.653515 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39510: BlockingRecv error: recv got EOF from 127.0.0.1:39510 (error 108)
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":355,"threads_started":1}
W20260926 07:33:56.654044 23413 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.655650 23413 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.656946 23413 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:56.671113 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.658351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.22.221.65:40421)
0926 07:33:56.658554 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.658561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.658831 (+ 270us) server_negotiation.cc:407] Connection header received
0926 07:33:56.658958 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.658962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.659017 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.659094 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.660142 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.661180 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.662303 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.662831 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.662924 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.666241 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.666250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.666263 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.666292 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.668831 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.669384 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.669390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.669393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.669503 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.669811 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.669813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.669815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.670129 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.670332 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.670599 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.670856 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:33:56.676400 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:56.674743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.22.221.65:40421)
0926 07:33:56.674886 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.674889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.675298 (+ 409us) server_negotiation.cc:407] Connection header received
0926 07:33:56.675442 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.675446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.675506 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.675578 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.676138 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39528: BlockingRecv error: recv got EOF from 127.0.0.1:39528 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:33:56.676807 23413 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc)
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.678174 23413 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.679468 23413 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:56.692677 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.680702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.22.221.65:40421)
0926 07:33:56.680869 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.680877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.681003 (+ 126us) server_negotiation.cc:407] Connection header received
0926 07:33:56.681221 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.681224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.681287 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.681356 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.682218 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.682868 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.684017 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.684417 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.684507 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.687730 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.687735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.687738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.687774 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.690587 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.691135 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.691138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.691140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.691317 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.691626 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.691629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.691631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.691861 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.692048 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.692230 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.692366 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 07:33:56.696588 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:56.695031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.22.221.65:40421)
0926 07:33:56.695222 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.695245 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.695440 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:33:56.695597 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.695603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.695681 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.695778 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.696374 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39550: BlockingRecv error: recv got EOF from 127.0.0.1:39550 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 07:33:56.696830 23413 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407989013554-23413-0/krb5kdc/krb5cc)
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.698254 23413 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:56.699515 23413 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:33:56.712622 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.700641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.22.221.65:40421)
0926 07:33:56.700836 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.700844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.701101 (+ 257us) server_negotiation.cc:407] Connection header received
0926 07:33:56.701251 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.701256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.701319 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.701412 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.702330 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.703011 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.704233 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.704642 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.704772 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.707932 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.707938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.707942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.707980 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.710228 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.710817 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.710823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.710827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.710978 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.711381 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.711390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.711392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.711621 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.711928 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.712150 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.712353 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:33:56.714030 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29096
I20260926 07:33:56.724757 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.714618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.22.221.65:40421)
0926 07:33:56.714811 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.714817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.715031 (+ 214us) server_negotiation.cc:407] Connection header received
0926 07:33:56.715147 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.715151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.715199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.715323 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.716123 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.716755 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.717838 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.718233 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.718318 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.720148 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.720152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.720155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.720182 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.722607 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.723144 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.723147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.723149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.723326 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.723612 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.723615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.723617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.723843 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.724025 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.724267 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.724441 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:33:56.725332 29219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:56.808202 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29096
I20260926 07:33:56.825435 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29230
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:33:56.846975 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.826550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.22.221.66:37467)
0926 07:33:56.827784 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.827795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.827826 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:33:56.828134 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.828147 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.828448 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.828651 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.829839 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.831451 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.833359 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.834147 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.834281 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.839847 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.839857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.839871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.839903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.843668 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.844314 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.844324 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.844329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.844481 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.844911 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.844920 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.844923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.845402 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.845715 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.846169 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.846483 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":902,"threads_started":1}
I20260926 07:33:56.848654 29353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:56.932415 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29230
I20260926 07:33:56.948158 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29363
Sep 26 07:33:56 dist-test-slave-qlq8 krb5kdc[28990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:33:56.963620 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.949222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.22.221.67:45657)
0926 07:33:56.949906 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:56.949912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:56.949939 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:33:56.949999 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:56.950005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:56.950209 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:56.950333 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:56.951420 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.952420 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.953617 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:56.954161 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:56.954255 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:56.958148 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:56.958159 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:56.958175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:56.958214 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:56.961081 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.961694 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.961700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.961702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.961824 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:56.962122 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:56.962125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:56.962127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:56.962445 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:56.962670 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:56.962859 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:33:56.963105 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":370,"threads_started":1}
I20260926 07:33:56.964864 29486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:57.048064 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29363
I20260926 07:33:57.064103 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29003
I20260926 07:33:57.065265 29065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:57.175796 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29003
2026-09-26T07:33:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3826 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:57 dist-test-slave-qlq8 krb5kdc[29518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:57 dist-test-slave-qlq8 krb5kdc[29518](info): set up 2 sockets
Sep 26 07:33:57 dist-test-slave-qlq8 krb5kdc[29518](info): commencing operation
krb5kdc: starting...
W20260926 07:33:58.260713 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:33:58 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab.
Entry for principal oryx/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab.
I20260926 07:33:58.467274 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42459
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab
--principal=oryx/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:58.789461 29531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:58.789731 29531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:58.789816 29531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:58.796916 29531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:58.797010 29531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:58.797058 29531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:58.797107 29531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:58.797143 29531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:58.806599 29531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:42459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab
--principal=oryx/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:58.807899 29531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:58.809134 29531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:58.820781 29536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:58.820842 29537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:58.820881 29539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:58.821789 29531 server_base.cc:1034] running on GCE node
I20260926 07:33:58.822495 29531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:58.823899 29531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:58.825121 29531 hybrid_clock.cc:648] HybridClock initialized: now 1790408038825069 us; error 57 us; skew 500 ppm
I20260926 07:33:58.825477 29531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:58 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:58.828511 29531 init.cc:377] Logged in from keytab as oryx/127.22.221.126@KRBTEST.COM (short username oryx)
I20260926 07:33:58.829941 29531 webserver.cc:460] Webserver started at http://127.22.221.126:40345/ using document root <none> and password file <none>
I20260926 07:33:58.830574 29531 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:58.830673 29531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:58.830912 29531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:58.833814 29531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "02fa98fbcd424706b89819a21d1e0290"
format_stamp: "Formatted at 2026-09-26 07:33:58 on dist-test-slave-qlq8"
server_key: "170b8f2ae72b3287f5169f0ba22c3c36"
server_key_iv: "8b290e507d8147dc0547125b6b47b8ef"
server_key_version: "encryptionkey@0"
I20260926 07:33:58.834507 29531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "02fa98fbcd424706b89819a21d1e0290"
format_stamp: "Formatted at 2026-09-26 07:33:58 on dist-test-slave-qlq8"
server_key: "170b8f2ae72b3287f5169f0ba22c3c36"
server_key_iv: "8b290e507d8147dc0547125b6b47b8ef"
server_key_version: "encryptionkey@0"
I20260926 07:33:58.839589 29531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:33:58.842737 29546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:58.844048 29531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:58.844218 29531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "02fa98fbcd424706b89819a21d1e0290"
format_stamp: "Formatted at 2026-09-26 07:33:58 on dist-test-slave-qlq8"
server_key: "170b8f2ae72b3287f5169f0ba22c3c36"
server_key_iv: "8b290e507d8147dc0547125b6b47b8ef"
server_key_version: "encryptionkey@0"
I20260926 07:33:58.844405 29531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:58.859764 29531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:58.863521 29531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:58.863785 29531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:58.881865 29531 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:42459
I20260926 07:33:58.881897 29597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:42459 every 8 connection(s)
I20260926 07:33:58.883246 29531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:58.886809 29598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:58.889520 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29531
I20260926 07:33:58.889731 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:33:58.890486 23413 external_mini_cluster.cc:1428] Setting key 3d21a500cd0118addf3cb5218806161c
I20260926 07:33:58.899441 29598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Bootstrap starting.
I20260926 07:33:58.903124 29598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:58.904456 29598 log.cc:824] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:33:58 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:33:58.907477 29598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: No bootstrap required, opened a new log
I20260926 07:33:58.913334 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:58.892617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.22.221.126:42459)
0926 07:33:58.893407 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:58.893418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:58.893458 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:33:58.895067 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:58.895095 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:58.895561 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:58.895890 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:58.897625 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:58.898895 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:58.900627 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:58.901294 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:58.901426 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:58.906276 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:58.906283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:58.906297 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:58.906329 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:58.909373 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:58.910074 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:58.910084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:58.910100 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:58.910264 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:58.910641 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:58.910647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:58.910650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:58.911154 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:58.911547 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:58.911933 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 07:33:58.912348 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":396,"threads_started":1}
I20260926 07:33:58.915333 29598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } }
I20260926 07:33:58.915729 29598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:58.915841 29598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fa98fbcd424706b89819a21d1e0290, State: Initialized, Role: FOLLOWER
I20260926 07:33:58.916426 29598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } }
I20260926 07:33:58.916692 29598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:58.916829 29598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:58.917001 29598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:58.919126 29598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } }
I20260926 07:33:58.919620 29598 leader_election.cc:302] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02fa98fbcd424706b89819a21d1e0290; no voters:
I20260926 07:33:58.920171 29598 leader_election.cc:288] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:58.920293 29603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:58.920981 29603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 LEADER]: Becoming Leader. State: Replica: 02fa98fbcd424706b89819a21d1e0290, State: Running, Role: LEADER
I20260926 07:33:58.921453 29603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } }
I20260926 07:33:58.922111 29598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:58.924485 29605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fa98fbcd424706b89819a21d1e0290. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } }
I20260926 07:33:58.924932 29605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:58.929805 29604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } }
I20260926 07:33:58.930145 29604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:58.930689 29611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:58.935351 29611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:58.944139 29611 catalog_manager.cc:1347] Generated new cluster ID: c259eddfa4434248a44bffc58578daae
I20260926 07:33:58.944257 29611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:58.960355 29611 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:58.961637 29611 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:58.967731 29611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Generated new TSK 0
I20260926 07:33:58.968672 29611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab.
Entry for principal oryx/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab.
I20260926 07:33:59.027457 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42459
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:59.349555 29624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:59.349915 29624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:59.350095 29624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:59.357515 29624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:59.357617 29624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:59.357728 29624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:33:59.367215 29624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:59.368461 29624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:59.369709 29624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:59.383152 29632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:59.383126 29629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:59.383206 29630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:59.383985 29624 server_base.cc:1034] running on GCE node
I20260926 07:33:59.384600 29624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:59.385658 29624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:59.386855 29624 hybrid_clock.cc:648] HybridClock initialized: now 1790408039386795 us; error 58 us; skew 500 ppm
I20260926 07:33:59.387177 29624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:59 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408039, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:59.390262 29624 init.cc:377] Logged in from keytab as oryx/127.22.221.65@KRBTEST.COM (short username oryx)
I20260926 07:33:59.391788 29624 webserver.cc:460] Webserver started at http://127.22.221.65:40825/ using document root <none> and password file <none>
I20260926 07:33:59.392418 29624 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:59.392516 29624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:59.392760 29624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:59.395732 29624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "64229b9e03ab45018198fe4555158212"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "83b235f66b7d388414f9a6328b0fe09d"
server_key_iv: "7c1c62632058e451945a942d31a9848b"
server_key_version: "encryptionkey@0"
I20260926 07:33:59.396482 29624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "64229b9e03ab45018198fe4555158212"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "83b235f66b7d388414f9a6328b0fe09d"
server_key_iv: "7c1c62632058e451945a942d31a9848b"
server_key_version: "encryptionkey@0"
I20260926 07:33:59.401309 29624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:33:59.404526 29639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:59.405771 29624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:59.405973 29624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "64229b9e03ab45018198fe4555158212"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "83b235f66b7d388414f9a6328b0fe09d"
server_key_iv: "7c1c62632058e451945a942d31a9848b"
server_key_version: "encryptionkey@0"
I20260926 07:33:59.406140 29624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:59.433763 29624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:59.437747 29624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:59.437987 29624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:59.438849 29624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:59.440192 29624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:59.440264 29624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:59.440344 29624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:59.440388 29624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:59.473076 29624 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:40833
I20260926 07:33:59.473120 29751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:40833 every 8 connection(s)
I20260926 07:33:59.474357 29624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:59.477308 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29624
I20260926 07:33:59.477515 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:59.478418 23413 external_mini_cluster.cc:1428] Setting key a9981fdc415712ae3ed38c18a125cab7
Sep 26 07:33:59 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408039, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:33:59.495925 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.476833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:58691 (local address 127.22.221.126:42459)
0926 07:33:59.477336 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:59.477341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:59.478414 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 07:33:59.480612 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:59.480616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:59.480676 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:59.480765 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:59.482833 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.483712 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.486180 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.486556 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.486662 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:59.490248 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:59.490253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:59.490256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:59.490286 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:59.492876 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:59.493580 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:59.493588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:59.493593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:59.493718 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:59.494137 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:59.494145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:59.494148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:59.494410 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:59.494563 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:59.495353 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 07:33:59.495575 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260926 07:33:59.496919 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.477306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:58691)
0926 07:33:59.478216 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:59.478278 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:59.480300 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:59.481039 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:59.481051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:59.481835 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:59.482666 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:59.482692 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.483874 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.483881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:59.486025 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:59.486048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.486792 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.486801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:59.487123 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:59.487919 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:59.487940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:59.490030 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:59.493018 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:59.493026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:59.493042 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:59.493455 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:59.493829 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:59.493834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:59.493838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:59.494021 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:59.494547 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:59.494554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:59.494890 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 07:33:59.495241 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 07:33:59.495791 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":343,"threads_started":1}
I20260926 07:33:59.498657 29752 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42459
I20260926 07:33:59.498960 29752 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:33:59.499742 29752 heartbeater.cc:511] Master 127.22.221.126:42459 requested a full tablet report, sending...
I20260926 07:33:59.501868 29563 ts_manager.cc:194] Registered new tserver with Master: 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.66@KRBTEST.COM" created.
I20260926 07:33:59.503468 29563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:58691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab.
Entry for principal oryx/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab.
I20260926 07:33:59.526410 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42459
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:59.843161 29758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:59.843591 29758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:59.843797 29758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:59.851080 29758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:59.851205 29758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:59.851432 29758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:33:59.861035 29758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:59.862207 29758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:59.863569 29758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:59.876106 29764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:59.876148 29766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:59.876173 29763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:59.876852 29758 server_base.cc:1034] running on GCE node
I20260926 07:33:59.877471 29758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:59.878332 29758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:59.879539 29758 hybrid_clock.cc:648] HybridClock initialized: now 1790408039879493 us; error 43 us; skew 500 ppm
I20260926 07:33:59.879844 29758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:59 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408039, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:59.883086 29758 init.cc:377] Logged in from keytab as oryx/127.22.221.66@KRBTEST.COM (short username oryx)
I20260926 07:33:59.884608 29758 webserver.cc:460] Webserver started at http://127.22.221.66:37105/ using document root <none> and password file <none>
I20260926 07:33:59.885326 29758 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:59.885463 29758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:59.885766 29758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:59.888599 29758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "09c9aa2ed60d443c99b34d681bad4294"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "d87e69f049e0f617624b31aa17023585"
server_key_iv: "72a8954b0b43ca5d6fdc8137c413e38f"
server_key_version: "encryptionkey@0"
I20260926 07:33:59.889355 29758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "09c9aa2ed60d443c99b34d681bad4294"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "d87e69f049e0f617624b31aa17023585"
server_key_iv: "72a8954b0b43ca5d6fdc8137c413e38f"
server_key_version: "encryptionkey@0"
I20260926 07:33:59.894418 29758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 07:33:59.897764 29773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:59.899004 29758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:33:59.899164 29758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "09c9aa2ed60d443c99b34d681bad4294"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "d87e69f049e0f617624b31aa17023585"
server_key_iv: "72a8954b0b43ca5d6fdc8137c413e38f"
server_key_version: "encryptionkey@0"
I20260926 07:33:59.899430 29758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:59.915129 29758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:59.919322 29758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:59.919608 29758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:59.920483 29758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:59.921738 29758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:59.921811 29758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:59.921983 29758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:59.922052 29758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:59.954945 29758 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:35679
I20260926 07:33:59.954986 29885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:35679 every 8 connection(s)
I20260926 07:33:59.956316 29758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:59.956583 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29758
I20260926 07:33:59.956764 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:59.957474 23413 external_mini_cluster.cc:1428] Setting key f25443da63cadc3d48611b803d281faf
Sep 26 07:33:59 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408039, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:33:59.977308 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.958983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:58523 (local address 127.22.221.126:42459)
0926 07:33:59.959162 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:59.959167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:59.960598 (+ 1431us) server_negotiation.cc:407] Connection header received
0926 07:33:59.962364 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:59.962373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:59.962466 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:59.962577 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:59.964091 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.965246 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.966789 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.967195 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.967362 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:59.971337 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:59.971343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:59.971346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:59.971377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:59.974017 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:59.974948 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:59.974953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:59.974957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:59.975097 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:59.975558 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:59.975565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:59.975568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:59.975849 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:59.976007 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:59.976741 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 07:33:59.976963 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:33:59.978240 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.959604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:58523)
0926 07:33:59.960419 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:59.960464 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:59.962099 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:59.962776 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:59.962783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:59.963347 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:59.963918 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:59.963935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.965392 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.965396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:59.966662 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:59.966679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:59.967451 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:59.967460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:59.967806 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:59.968492 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:59.968525 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:59.971093 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:59.974203 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:59.974212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:59.974228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:59.974820 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:59.975220 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:59.975225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:59.975248 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:59.975443 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:59.976080 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:59.976087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:59.976382 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 07:33:59.976651 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 07:33:59.977145 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":339,"threads_started":1}
WARNING: no policy specified for oryx/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:33:59.979655 29886 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42459
I20260926 07:33:59.979957 29886 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:59.980947 29886 heartbeater.cc:511] Master 127.22.221.126:42459 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.67@KRBTEST.COM" created.
I20260926 07:33:59.982692 29563 ts_manager.cc:194] Registered new tserver with Master: 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679)
I20260926 07:33:59.983553 29563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:58523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab.
Entry for principal oryx/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab.
I20260926 07:34:00.005754 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42459
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:00.322763 29891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:00.323031 29891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:00.323127 29891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:00.330170 29891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:00.330268 29891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:00.330384 29891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:00.339370 29891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:00.340508 29891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:00.341696 29891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:00.355311 29896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:00.355216 29897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:00.355295 29899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:00.356549 29891 server_base.cc:1034] running on GCE node
I20260926 07:34:00.357160 29891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:00.357962 29891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:00.359161 29891 hybrid_clock.cc:648] HybridClock initialized: now 1790408040359113 us; error 43 us; skew 500 ppm
I20260926 07:34:00.359469 29891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:00 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408040, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:00.362481 29891 init.cc:377] Logged in from keytab as oryx/127.22.221.67@KRBTEST.COM (short username oryx)
I20260926 07:34:00.363852 29891 webserver.cc:460] Webserver started at http://127.22.221.67:41417/ using document root <none> and password file <none>
I20260926 07:34:00.364490 29891 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:00.364629 29891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:00.364925 29891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:00.367774 29891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "38384ae8c9ad4ebc8b0e04c128a56039"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "299ed11665ace6c13069421c4114f205"
server_key_iv: "190323c6a30aca6f7f987ea2c8ea5e3f"
server_key_version: "encryptionkey@0"
I20260926 07:34:00.368551 29891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "38384ae8c9ad4ebc8b0e04c128a56039"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "299ed11665ace6c13069421c4114f205"
server_key_iv: "190323c6a30aca6f7f987ea2c8ea5e3f"
server_key_version: "encryptionkey@0"
I20260926 07:34:00.373432 29891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 07:34:00.376742 29906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:00.377971 29891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:34:00.378144 29891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "38384ae8c9ad4ebc8b0e04c128a56039"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "299ed11665ace6c13069421c4114f205"
server_key_iv: "190323c6a30aca6f7f987ea2c8ea5e3f"
server_key_version: "encryptionkey@0"
I20260926 07:34:00.378342 29891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:00.400386 29891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:00.404790 29891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:00.405076 29891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:00.405982 29891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:00.407294 29891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:00.407398 29891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:00.407563 29891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:00.407639 29891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:00.440332 29891 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:32773
I20260926 07:34:00.440414 30018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:32773 every 8 connection(s)
I20260926 07:34:00.441576 29891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:00.443617 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 29891
I20260926 07:34:00.443794 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:00.444350 23413 external_mini_cluster.cc:1428] Setting key 03b4fb3c4f86cceb1a4368366b3ed82f
Sep 26 07:34:00 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408040, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:00.460662 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.444108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:40505 (local address 127.22.221.126:42459)
0926 07:34:00.444310 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:00.444316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:00.445595 (+ 1279us) server_negotiation.cc:407] Connection header received
0926 07:34:00.447466 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:00.447470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:00.447563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:00.447643 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:00.449041 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.449771 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.451181 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.451643 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.451734 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.454887 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:00.454892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:00.454895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:00.454924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:00.457673 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.458502 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.458507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.458510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.458641 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.459093 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.459096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.459098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.459507 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:00.459643 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:00.460268 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 07:34:00.460421 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:34:00.461589 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.444648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:40505)
0926 07:34:00.445448 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:00.445491 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:00.447173 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:00.447841 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:00.447849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:00.448320 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:00.448903 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:00.448922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.449891 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.449894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:00.451054 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:00.451066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.451822 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.451827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:00.452012 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.452588 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:00.452609 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:00.454693 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:00.457819 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:00.457827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:00.457840 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:00.458374 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:00.458751 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:00.458755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:00.458759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:00.458964 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:00.459648 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:00.459654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:00.459942 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 07:34:00.460175 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 07:34:00.460528 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
I20260926 07:34:00.463940 30019 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42459
I20260926 07:34:00.464257 30019 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:00.464937 30019 heartbeater.cc:511] Master 127.22.221.126:42459 requested a full tablet report, sending...
I20260926 07:34:00.466486 29563 ts_manager.cc:194] Registered new tserver with Master: 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773)
I20260926 07:34:00.467679 29563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:40505
I20260926 07:34:00.472015 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.125@KRBTEST.COM" created.
I20260926 07:34:00.507093 29752 heartbeater.cc:503] Master 127.22.221.126:42459 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.125.keytab.
Entry for principal oryx/127.22.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.125.keytab.
I20260926 07:34:00.518069 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:44963
--webserver_interface=127.22.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.125.keytab
--principal=oryx/127.22.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:42459,127.22.221.125:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:00.832065 30024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:00.832404 30024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:00.832569 30024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:00.839742 30024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:00.839869 30024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:00.839979 30024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:00.840073 30024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:00.840153 30024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:00.849731 30024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:42459,127.22.221.125:44963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.125.keytab
--principal=oryx/127.22.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
W20260926 07:34:00.850085 30024 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.221.126:42459,127.22.221.125:44963'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 07:34:00.851095 30024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:00.852425 30024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:00.863904 30029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:00.863873 30030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:00.864140 30032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:00.864620 30024 server_base.cc:1034] running on GCE node
I20260926 07:34:00.865290 30024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:00.866536 30024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:00.867758 30024 hybrid_clock.cc:648] HybridClock initialized: now 1790408040867706 us; error 29 us; skew 500 ppm
I20260926 07:34:00.868026 30024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:00 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408040, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:00.871037 30024 init.cc:377] Logged in from keytab as oryx/127.22.221.125@KRBTEST.COM (short username oryx)
I20260926 07:34:00.872357 30024 webserver.cc:460] Webserver started at http://127.22.221.125:43913/ using document root <none> and password file <none>
I20260926 07:34:00.873034 30024 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:00.873144 30024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:00.873456 30024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:00.876317 30024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/instance:
uuid: "940f3dedb10544cea8abc0ba623aef3e"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "ef8f514a1b74fcd103c69333b54a67ef"
server_key_iv: "b533e6184d1b8b6ac4350d83a5c5c863"
server_key_version: "encryptionkey@0"
I20260926 07:34:00.877094 30024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal/instance:
uuid: "940f3dedb10544cea8abc0ba623aef3e"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "ef8f514a1b74fcd103c69333b54a67ef"
server_key_iv: "b533e6184d1b8b6ac4350d83a5c5c863"
server_key_version: "encryptionkey@0"
I20260926 07:34:00.882167 30024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:34:00.885689 30039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:00.887028 30024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:34:00.887197 30024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
uuid: "940f3dedb10544cea8abc0ba623aef3e"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "ef8f514a1b74fcd103c69333b54a67ef"
server_key_iv: "b533e6184d1b8b6ac4350d83a5c5c863"
server_key_version: "encryptionkey@0"
I20260926 07:34:00.887452 30024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:00.897949 30024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:00.901746 30024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:00.902050 30024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:00.919353 30024 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.125:44963
I20260926 07:34:00.919371 30090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.125:44963 every 8 connection(s)
I20260926 07:34:00.920671 30024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
I20260926 07:34:00.924235 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:00.927702 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 30024
I20260926 07:34:00.930950 30091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:34:00 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
Sep 26 07:34:00 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408040, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.125@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:00.949875 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.934950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.22.221.126:42459)
0926 07:34:00.935166 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:00.935174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:00.935919 (+ 745us) server_negotiation.cc:407] Connection header received
0926 07:34:00.936083 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:00.936089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:00.936164 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:00.936263 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:00.937278 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.937997 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.939458 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.939856 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.940005 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.943509 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:00.943514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:00.943517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:00.943548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:00.947069 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.947814 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.947820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.947824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.948000 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.948399 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.948404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.948408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.948717 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:00.948872 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:00.949434 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 07:34:00.949646 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:34:00.950603 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.929818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.22.221.125:44963)
0926 07:34:00.930581 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:00.930596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:00.930643 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:34:00.932352 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:00.932380 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:00.932902 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:00.933322 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:00.934880 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.936592 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.937839 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.938486 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.938674 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.943102 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:00.943113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:00.943127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:00.943163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:00.946429 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.947089 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.947096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.947104 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.947326 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.947691 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.947698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.947704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.948289 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:00.948590 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:00.949074 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 07:34:00.949506 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":331,"threads_started":1}
I20260926 07:34:00.950626 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.934952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:48876)
0926 07:34:00.935768 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:00.935818 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:00.935956 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:00.936312 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:00.936316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:00.936903 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:00.937127 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:00.937135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.938171 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.938176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:00.939320 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:00.939332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.940080 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.940088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:00.940263 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.940847 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:00.940867 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:00.943318 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:00.947210 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:00.947223 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:00.947257 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:00.947682 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:00.948132 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:00.948137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:00.948141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:00.948283 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:00.948864 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:00.948871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:00.949098 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 07:34:00.950068 (+ 970us) client_negotiation.cc:240] Negotiation successful
0926 07:34:00.950301 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"spinlock_wait_cycles":13568,"thread_start_us":418,"threads_started":1}
I20260926 07:34:00.953275 30091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:34:00 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408040, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.125@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
I20260926 07:34:00.965322 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.953940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:37194)
0926 07:34:00.954077 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:00.954091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:00.954198 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:00.954471 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:00.954474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:00.954803 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:00.954922 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:00.954930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.956006 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.956010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:00.956847 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:00.956857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.957489 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.957493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:00.957692 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.958208 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:00.958223 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:00.960097 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:00.962987 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:00.962994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:00.962997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:00.963424 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:00.963766 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:00.963772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:00.963776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:00.963906 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:00.964380 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:00.964385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:00.964536 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:34:00.964673 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 07:34:00.964945 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260926 07:34:00.965355 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.954099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.22.221.125:44963)
0926 07:34:00.954248 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:00.954256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:00.954272 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:00.954317 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:00.954320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:00.954364 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:00.954455 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:00.955136 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.955868 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.956978 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:00.957355 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:00.957482 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:00.960269 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:00.960274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:00.960278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:00.960303 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:00.962861 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.963533 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.963540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.963544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.963655 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:00.964004 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:00.964011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:00.964014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:00.964273 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:00.964416 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:00.964742 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:34:00.965060 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":57}
I20260926 07:34:00.970008 30091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Bootstrap starting.
I20260926 07:34:00.972901 30091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:00.973809 30091 log.cc:824] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:00.976243 30091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: No bootstrap required, opened a new log
I20260926 07:34:00.983155 30091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } }
I20260926 07:34:00.983494 30091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:00.983572 30091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940f3dedb10544cea8abc0ba623aef3e, State: Initialized, Role: FOLLOWER
I20260926 07:34:00.984025 30091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } }
I20260926 07:34:00.984726 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } } }
I20260926 07:34:00.985038 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:00.985635 30091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:00.986920 29886 heartbeater.cc:503] Master 127.22.221.126:42459 was elected leader, sending a full tablet report...
W20260926 07:34:00.994139 30108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:00.994264 30108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:00.996589 30024 master_runner.cc:428] Detected that this master 940f3dedb10544cea8abc0ba623aef3e is joining an existing cluster
I20260926 07:34:00.996687 30024 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.221.125:44963
I20260926 07:34:01.016779 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.000479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:48882)
0926 07:34:01.001158 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.001180 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.001348 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.001677 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.001681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.002214 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.002638 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.002653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.003901 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.003908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.005995 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.006007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.007152 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.007158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.007333 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.008003 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.008026 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.009151 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.013255 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.013260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.013263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.013823 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.014303 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.014313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.014317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.014500 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.015518 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.015524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.015744 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.015904 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.016269 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":1,"thread_start_us":406,"threads_started":1}
I20260926 07:34:01.017792 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.000401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.22.221.126:42459)
0926 07:34:01.000595 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.000604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.001284 (+ 680us) server_negotiation.cc:407] Connection header received
0926 07:34:01.001465 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.001470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.001551 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.001659 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.002781 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.003772 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.006167 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.006797 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.006919 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.009419 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.009426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.009431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.009469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.013094 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.013976 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.013982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.013988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.014149 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.014945 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.014952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.014955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.015364 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.017075 (+ 1711us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.017215 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.017441 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:34:01.020412 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.000959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.22.221.125:44963)
0926 07:34:01.001141 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.001146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.004654 (+ 3508us) server_negotiation.cc:407] Connection header received
0926 07:34:01.004819 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.004825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.004896 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.004994 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.006343 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.007315 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.009296 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.009999 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.010097 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.012497 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.012504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.012511 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.012552 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.016472 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.017422 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.017427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.017434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.017547 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.018445 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.018450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.018453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.018799 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.019429 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.019889 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.020084 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:34:01.021555 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.000580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:37210)
0926 07:34:01.004541 (+ 3961us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.004558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.004698 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.005155 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.005158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.005851 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.006199 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.006213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.007522 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.007527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.009149 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.009159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.010416 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.010425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.010610 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.011155 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.011182 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.012302 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.016630 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.016635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.016639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.017308 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.017657 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.017663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.017666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.017800 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.019482 (+ 1682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.019488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.019636 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.020810 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.021094 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3835,"thread_start_us":2880,"threads_started":1}
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.126@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.22.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.124@KRBTEST.COM" created.
I20260926 07:34:01.039121 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.024097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:37224)
0926 07:34:01.024900 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.024933 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.025085 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.025598 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.025605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.026186 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.026378 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.026387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.027409 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.027414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.028549 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.028561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.029331 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.029337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.029535 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.030127 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.030144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.032415 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.036099 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.036111 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.036118 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.036673 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.037126 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.037132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.037136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.037295 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.037948 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.037957 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.038158 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.038384 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.038693 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":482,"threads_started":1}
I20260926 07:34:01.039217 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.024451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.22.221.125:44963)
0926 07:34:01.025263 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.025268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.025283 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:34:01.025361 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.025364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.025434 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.025549 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.026530 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.027276 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.028698 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.029165 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.029303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.032580 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.032584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.032587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.032680 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.035948 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.036808 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.036817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.036823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.036990 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.037422 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.037430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.037436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.037798 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.037954 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.038497 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.038751 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698}
I20260926 07:34:01.041087 29563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.221.125:44963: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e"
member_type: NON_VOTER
last_known_addr {
host: "127.22.221.125"
port: 44963
}
attrs {
promote: true
}
}
dest_uuid: "02fa98fbcd424706b89819a21d1e0290"
cas_config_opid_index: -1
I20260926 07:34:01.042284 29563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } }
I20260926 07:34:01.058706 30066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:01.058414 30121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 940f3dedb10544cea8abc0ba623aef3e (127.22.221.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } }
I20260926 07:34:01.062383 30066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Refusing update from remote peer 02fa98fbcd424706b89819a21d1e0290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 07:34:01.063333 30121 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.221.125:44963
I20260926 07:34:01.063796 30121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } }
I20260926 07:34:01.064026 30121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:01.063865 30120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.124.keytab.
Entry for principal oryx/127.22.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.124.keytab.
I20260926 07:34:01.068588 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:44905
--webserver_interface=127.22.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.124.keytab
--principal=oryx/127.22.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:01.079357 30024 master.cc:537] Master@127.22.221.125:44963 shutting down...
W20260926 07:34:01.082275 29549 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 07:34:01.093899 30024 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:01.094331 30024 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:01.094529 30024 tablet_replica.cc:330] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: stopping tablet replica
W20260926 07:34:01.095053 29549 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 -> Peer 940f3dedb10544cea8abc0ba623aef3e (127.22.221.125:44963): Couldn't send request to peer 940f3dedb10544cea8abc0ba623aef3e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:34:01.100924 30024 master.cc:552] Master@127.22.221.125:44963 shutdown complete.
I20260926 07:34:01.101009 30024 master_runner.cc:305] Clearing existing system tablet
I20260926 07:34:01.101413 30024 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:01.101511 30024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:01.104389 30024 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 07:34:01.105978 30127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:01.106295 30024 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:01.106627 30024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:34:01.106755 30024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
uuid: "940f3dedb10544cea8abc0ba623aef3e"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "ef8f514a1b74fcd103c69333b54a67ef"
server_key_iv: "b533e6184d1b8b6ac4350d83a5c5c863"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.111701 30024 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:01.118724 30024 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:34:01.118932 30024 log.cc:1196] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 07:34:01.119772 30024 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Deleting consensus metadata
I20260926 07:34:01.120075 30024 master_runner.cc:315] Copying system tablet from 127.22.221.126:42459
I20260926 07:34:01.122579 30024 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.221.126:42459
I20260926 07:34:01.142038 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.123402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.22.221.126:42459)
0926 07:34:01.123612 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.123619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.127759 (+ 4140us) server_negotiation.cc:407] Connection header received
0926 07:34:01.127931 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.127937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.128038 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.128149 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.129157 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.130287 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.132193 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.132968 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.133122 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.135427 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.135436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.135443 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.135490 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.139172 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.139966 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.139972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.139975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.140125 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.140538 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.140542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.140545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.140899 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.141046 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.141489 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.141711 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:34:01.142457 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.123393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:48886)
0926 07:34:01.127623 (+ 4230us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.127644 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.127797 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.128191 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.128196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.128723 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.128981 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.128995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.130460 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.130466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.132039 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.132055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.133133 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.133140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.133337 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.134124 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.134148 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.135185 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.139320 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.139326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.139330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.139822 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.140243 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.140248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.140252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.140413 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.141028 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.141034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.141204 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.141347 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.141660 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4082,"thread_start_us":3984,"threads_started":1}
I20260926 07:34:01.143285 29583 tablet_copy_service.cc:140] P 02fa98fbcd424706b89819a21d1e0290: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 940f3dedb10544cea8abc0ba623aef3e ({username='oryx', principal='oryx/127.22.221.125@KRBTEST.COM'} at 127.0.0.1:48886)
I20260926 07:34:01.143601 29583 tablet_copy_service.cc:161] P 02fa98fbcd424706b89819a21d1e0290: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 940f3dedb10544cea8abc0ba623aef3e at {username='oryx', principal='oryx/127.22.221.125@KRBTEST.COM'} at 127.0.0.1:48886: session id = 940f3dedb10544cea8abc0ba623aef3e-00000000000000000000000000000000
I20260926 07:34:01.147919 29583 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:34:01.150007 30024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:34:01.156731 30024 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: tablet copy: Starting download of 0 data blocks...
I20260926 07:34:01.157034 30024 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: tablet copy: Starting download of 1 WAL segments...
I20260926 07:34:01.159648 30024 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:34:01.164034 29583 tablet_copy_service.cc:342] P 02fa98fbcd424706b89819a21d1e0290: Request end of tablet copy session 940f3dedb10544cea8abc0ba623aef3e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.221.125@KRBTEST.COM'} at 127.0.0.1:48886
I20260926 07:34:01.164232 29583 tablet_copy_service.cc:433] P 02fa98fbcd424706b89819a21d1e0290: ending tablet copy session 940f3dedb10544cea8abc0ba623aef3e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 940f3dedb10544cea8abc0ba623aef3e
I20260926 07:34:01.176039 30024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:01.179162 30135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:01.179437 30134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:01.184552 30024 server_base.cc:1034] running on GCE node
W20260926 07:34:01.184820 30137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:01.185241 30024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:01.185904 30024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:01.187129 30024 hybrid_clock.cc:648] HybridClock initialized: now 1790408041187059 us; error 36 us; skew 500 ppm
I20260926 07:34:01.187354 30024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408041, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:01.189522 30024 init.cc:377] Logged in from keytab as oryx/127.22.221.125@KRBTEST.COM (short username oryx)
I20260926 07:34:01.190665 30024 webserver.cc:460] Webserver started at http://127.22.221.125:34571/ using document root <none> and password file <none>
I20260926 07:34:01.191018 30024 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:01.191104 30024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:01.194399 30024 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:34:01.196341 30144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:01.196784 30024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:34:01.196924 30024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
uuid: "940f3dedb10544cea8abc0ba623aef3e"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "ef8f514a1b74fcd103c69333b54a67ef"
server_key_iv: "b533e6184d1b8b6ac4350d83a5c5c863"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.197100 30024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:01.218077 30024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:01.222101 30024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:01.247542 30024 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.125:44963
I20260926 07:34:01.247591 30195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.125:44963 every 8 connection(s)
I20260926 07:34:01.248647 30024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
I20260926 07:34:01.253801 30196 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:34:01.257791 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Bootstrap starting.
I20260926 07:34:01.273765 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:34:01.274281 30196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: Bootstrap complete.
I20260926 07:34:01.275316 30196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } }
I20260926 07:34:01.275579 30196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 940f3dedb10544cea8abc0ba623aef3e, State: Initialized, Role: LEARNER
I20260926 07:34:01.275801 30196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } }
I20260926 07:34:01.276957 30196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:01.276741 30199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } }
I20260926 07:34:01.277248 30199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: LEARNER
I20260926 07:34:01.286435 30210 catalog_manager.cc:1259] Loaded cluster ID: c259eddfa4434248a44bffc58578daae
I20260926 07:34:01.286530 30210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: loading cluster ID for follower catalog manager: success
I20260926 07:34:01.289901 30210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: acquiring CA information for follower catalog manager: success
I20260926 07:34:01.292501 30210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:34:01.455592 30124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:01.455919 30124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:01.456064 30124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:01.463433 30124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:01.463532 30124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:01.463624 30124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:01.463717 30124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:01.463797 30124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:01.471056 30019 heartbeater.cc:503] Master 127.22.221.126:42459 was elected leader, sending a full tablet report...
I20260926 07:34:01.474231 30124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.124.keytab
--principal=oryx/127.22.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:44905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:01.475594 30124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:01.477010 30124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:01.489144 30215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:01.489163 30218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:01.489158 30216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:01.490010 30124 server_base.cc:1034] running on GCE node
I20260926 07:34:01.490727 30124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:01.492126 30124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:01.493371 30124 hybrid_clock.cc:648] HybridClock initialized: now 1790408041493314 us; error 40 us; skew 500 ppm
I20260926 07:34:01.493793 30124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408041, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:01.497825 30124 init.cc:377] Logged in from keytab as oryx/127.22.221.124@KRBTEST.COM (short username oryx)
I20260926 07:34:01.499936 30124 webserver.cc:460] Webserver started at http://127.22.221.124:36439/ using document root <none> and password file <none>
I20260926 07:34:01.500594 30124 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:01.500742 30124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:01.500768 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.493582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:37234)
0926 07:34:01.493769 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.493788 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.493910 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.494801 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.494804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.494868 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:01.495075 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.495089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.496957 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.496964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.498655 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.498667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.499990 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.499995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.500198 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.500248 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.500358 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.500489 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:34:01.500835 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.493787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.22.221.125:44963)
0926 07:34:01.494418 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.494427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.494451 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:34:01.494535 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.494539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.494638 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.494768 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:01.495250 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.496817 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.498794 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.499773 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.499987 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.500164 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.500383 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.500512 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":319,"threads_started":1}
I20260926 07:34:01.501163 30124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:01.501982 30171 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 07:34:01.502723 30199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fa98fbcd424706b89819a21d1e0290. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } }
I20260926 07:34:01.502987 30199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: LEARNER
I20260926 07:34:01.504772 30124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/instance:
uuid: "34d9ed909226465db4db83e3a38d6f3b"
format_stamp: "Formatted at 2026-09-26 07:34:01 on dist-test-slave-qlq8"
server_key: "c065037d2c9be4b2c150c722dc710f79"
server_key_iv: "405d9637e62d60ca30ac78b7791caafa"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.505534 30124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal/instance:
uuid: "34d9ed909226465db4db83e3a38d6f3b"
format_stamp: "Formatted at 2026-09-26 07:34:01 on dist-test-slave-qlq8"
server_key: "c065037d2c9be4b2c150c722dc710f79"
server_key_iv: "405d9637e62d60ca30ac78b7791caafa"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.510775 30124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:34:01.514016 30226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:01.515296 30124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:01.515463 30124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
uuid: "34d9ed909226465db4db83e3a38d6f3b"
format_stamp: "Formatted at 2026-09-26 07:34:01 on dist-test-slave-qlq8"
server_key: "c065037d2c9be4b2c150c722dc710f79"
server_key_iv: "405d9637e62d60ca30ac78b7791caafa"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.515650 30124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:01.536922 30124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:01.540657 30124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:01.540928 30124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:01.559154 30124 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.124:44905
I20260926 07:34:01.559188 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.124:44905 every 8 connection(s)
I20260926 07:34:01.560477 30124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
I20260926 07:34:01.561210 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 30124
I20260926 07:34:01.565176 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:01.573416 30278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408041, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.124@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.124@KRBTEST.COM
I20260926 07:34:01.590199 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.575906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.22.221.126:42459)
0926 07:34:01.576158 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.576165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.576551 (+ 386us) server_negotiation.cc:407] Connection header received
0926 07:34:01.576766 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.576770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.576830 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.576931 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.577974 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.578741 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.580091 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.580482 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.580617 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.584065 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.584070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.584075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.584106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.586879 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.587794 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.587798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.587800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.587909 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.588370 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.588378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.588381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.588752 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.588912 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.589685 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.589834 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:34:01.591168 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.575701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:48894)
0926 07:34:01.576432 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.576464 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.576634 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.576991 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.576999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.577603 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.577827 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.577835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.578877 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.578882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.579944 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.579956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.580700 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.580707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.580880 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.581543 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.581578 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.583885 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.587016 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.587025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.587034 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.587690 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.588016 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.588022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.588026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.588248 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.588901 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.588907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.589288 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.589596 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.590051 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":399,"threads_started":1}
I20260926 07:34:01.591190 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.563278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.22.221.124:44905)
0926 07:34:01.564136 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.564151 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.564198 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:34:01.566440 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.566464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.566922 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.567353 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.568625 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.570628 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.571913 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.572896 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.573073 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.585141 (+ 12068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.585148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.585158 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.585188 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.588153 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.588924 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.588931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.588933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.589097 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.589455 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.589459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.589462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.589826 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.590113 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.590495 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.590667 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":374,"threads_started":1}
I20260926 07:34:01.592710 30278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408041, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.124@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
I20260926 07:34:01.604423 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.593392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:37248)
0926 07:34:01.593589 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.593608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.593704 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.594081 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.594085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.594528 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.594656 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.594664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.595673 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.595676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.596540 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.596551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.597219 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.597223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.597358 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.597944 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.597960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.599810 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.602444 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.602447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.602450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.602786 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.603113 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.603116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.603118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.603295 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.603773 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.603776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.603881 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.603984 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.604221 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:34:01.604432 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.593561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.22.221.125:44963)
0926 07:34:01.593767 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.593774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.593793 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:01.593852 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.593857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.593936 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.594042 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.594790 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.595554 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.596664 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.597108 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.597198 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.599981 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.599985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.599988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.600018 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.602311 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.602880 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.602883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.602885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.602996 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.603395 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.603398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.603400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.603676 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.603784 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.604071 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.604224 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:34:01.605273 30278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408041, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.124@KRBTEST.COM for oryx/127.22.221.124@KRBTEST.COM
I20260926 07:34:01.616379 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.605799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:40432)
0926 07:34:01.605967 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.606003 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.606102 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.606377 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.606381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.606700 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.606821 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.606828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.607795 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.607803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.608588 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.608599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.609248 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.609253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.609498 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.609965 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.609980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.611736 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.614421 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.614424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.614426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.614750 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.615057 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.615060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.615062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.615197 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.615730 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.615733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.615849 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.615954 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.616099 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":3}
I20260926 07:34:01.616477 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.605952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.22.221.124:44905)
0926 07:34:01.606090 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.606096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.606112 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:01.606214 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.606219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.606274 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.606363 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.606945 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.607681 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.608721 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.609122 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.609242 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.611894 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.611899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.611902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.611928 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.614313 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.614843 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.614846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.614848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.614960 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.615373 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.615382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.615385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.615631 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.615736 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.616036 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.616204 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":49}
I20260926 07:34:01.621037 30278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Bootstrap starting.
I20260926 07:34:01.624126 30278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:01.625093 30278 log.cc:824] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:01.627503 30278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: No bootstrap required, opened a new log
I20260926 07:34:01.634410 30278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } }
I20260926 07:34:01.634706 30278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:01.634816 30278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d9ed909226465db4db83e3a38d6f3b, State: Initialized, Role: FOLLOWER
I20260926 07:34:01.635368 30278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } }
I20260926 07:34:01.636140 30284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } } }
I20260926 07:34:01.636409 30284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:01.637055 30278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:34:01.645720 30295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:01.645843 30295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:01.646564 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29624
I20260926 07:34:01.649840 30124 master_runner.cc:428] Detected that this master 34d9ed909226465db4db83e3a38d6f3b is joining an existing cluster
I20260926 07:34:01.649966 30124 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.221.124:44905
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.65@KRBTEST.COM
I20260926 07:34:01.670048 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.654124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.22.221.125:44963)
0926 07:34:01.654292 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.654298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.654929 (+ 631us) server_negotiation.cc:407] Connection header received
0926 07:34:01.655164 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.655172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.655272 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.655339 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.656882 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.657955 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.659967 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.660649 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.660746 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.663299 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.663307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.663314 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.663356 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.666834 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.667630 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.667635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.667638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.667783 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.668261 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.668266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.668270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.668609 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.668741 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.669418 (+ 677us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.669661 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:34:01.670994 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.654202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:37254)
0926 07:34:01.654832 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.654851 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.655027 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.655554 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.655559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.656322 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.656712 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.656725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.658138 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.658143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.659800 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.659816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.661025 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.661031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.661206 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.661861 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.661884 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.663070 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.667000 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.667005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.667009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.667479 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.667935 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.667939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.667943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.668132 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.668842 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.668846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.669073 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.670233 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.670535 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":362,"threads_started":1}
I20260926 07:34:01.673764 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.647665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.22.221.65:40833)
0926 07:34:01.649111 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.649119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.649148 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:34:01.649216 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.649224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.649454 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.651350 (+ 1896us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.651611 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.653228 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.655139 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.656267 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.656419 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.663458 (+ 7039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.663473 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.663492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.663545 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.668351 (+ 4806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.669323 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.669335 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.669341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.669516 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.670038 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.670045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.670065 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.670815 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.672683 (+ 1868us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.672887 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.673215 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":1113,"threads_started":1}
I20260926 07:34:01.675608 29747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:01.685701 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.654045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.22.221.126:42459)
0926 07:34:01.654214 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.654220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.671301 (+ 17081us) server_negotiation.cc:407] Connection header received
0926 07:34:01.671534 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.671540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.671631 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.671706 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.672771 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.673899 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.675815 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.676486 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.676596 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.679010 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.679016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.679021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.679060 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.682429 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.683408 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.683414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.683417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.683617 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.684084 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.684090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.684093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.684418 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.684547 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.685137 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.685354 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:01.686676 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.654511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:48908)
0926 07:34:01.671193 (+ 16682us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.671212 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.671392 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.671906 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.671911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.672390 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.672598 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.672610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.674099 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.674104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.675642 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.675657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.676873 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.676882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.677070 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.677655 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.677679 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.678810 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.682632 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.682641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.682646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.683271 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.683774 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.683779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.683783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.683952 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.684644 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.684648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.684807 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.685942 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.686234 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16589,"thread_start_us":25294,"threads_started":1}
I20260926 07:34:01.703841 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.654968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:40434)
0926 07:34:01.679680 (+ 24712us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.679703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.680088 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.680616 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.680623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.681378 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.681784 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.681799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.685226 (+ 3427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.685234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.687973 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.687993 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.693143 (+ 5150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.693157 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.693407 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.695298 (+ 1891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.695325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.696454 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.700478 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.700483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.700487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.701071 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.701507 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.701513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.701517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.701660 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.702316 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.702322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.702597 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.702766 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.703119 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24533,"thread_start_us":31933,"threads_started":1}
I20260926 07:34:01.703869 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.655749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.22.221.124:44905)
0926 07:34:01.655938 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.655943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.679789 (+ 23846us) server_negotiation.cc:407] Connection header received
0926 07:34:01.680230 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.680243 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.680321 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.680395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.682425 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.683895 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.688317 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.689247 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.689371 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.696648 (+ 7277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.696655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.696659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.696700 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.700335 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.701228 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.701234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.701237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.701373 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.701806 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.701811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.701815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.702162 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.702404 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.702918 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.703309 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":60}
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.126@KRBTEST.COM for oryx/127.22.221.124@KRBTEST.COM
I20260926 07:34:01.721771 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.693823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:40446)
0926 07:34:01.694069 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.694088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.694186 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.695117 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.695125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.695631 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.695825 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.695838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.709103 (+ 13265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.709109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.710582 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.710598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.711625 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.711631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.711802 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.712407 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.712429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.715105 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.719022 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.719028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.719032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.719554 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.719974 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.719979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.719983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.720164 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.720790 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.720797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.720956 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.721113 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.721387 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:34:01.721798 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.694155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.22.221.124:44905)
0926 07:34:01.694690 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.694696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.694718 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:01.694804 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.694813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.694888 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.695044 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.707631 (+ 12587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.708938 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.710738 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.711450 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.711601 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.715363 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.715370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.715375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.715409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.718894 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.719702 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.719708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.719712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.719865 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.720283 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.720288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.720291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.720653 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.720779 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.721207 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.721399 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
I20260926 07:34:01.723616 29563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.221.124:44905: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b"
member_type: NON_VOTER
last_known_addr {
host: "127.22.221.124"
port: 44905
}
attrs {
promote: true
}
}
dest_uuid: "02fa98fbcd424706b89819a21d1e0290"
cas_config_opid_index: 5
I20260926 07:34:01.724601 29563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } }
I20260926 07:34:01.726394 30120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.730158 30120 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.221.124:44905
I20260926 07:34:01.730736 30124 master.cc:537] Master@127.22.221.124:44905 shutting down...
I20260926 07:34:01.734889 30171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 LEARNER]: Refusing update from remote peer 02fa98fbcd424706b89819a21d1e0290: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:34:01.744812 30124 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:01.745203 30124 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:01.745383 30124 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: stopping tablet replica
I20260926 07:34:01.748494 30120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 07:34:01.748708 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.749101 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
W20260926 07:34:01.753683 29549 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 -> Peer 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124:44905): Couldn't send request to peer 34d9ed909226465db4db83e3a38d6f3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:34:01.761678 30124 master.cc:552] Master@127.22.221.124:44905 shutdown complete.
I20260926 07:34:01.761765 30124 master_runner.cc:305] Clearing existing system tablet
I20260926 07:34:01.762122 30124 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:01.762225 30124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:01.768498 30171 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.771047 30124 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.009s
I20260926 07:34:01.773319 30199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: NON_VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: true } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.773597 30199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: LEARNER
I20260926 07:34:01.774842 30120 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: attempting to promote NON_VOTER 940f3dedb10544cea8abc0ba623aef3e to VOTER
I20260926 07:34:01.776003 30120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } }
I20260926 07:34:01.778286 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.777826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:40452)
0926 07:34:01.777979 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.778110 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:44905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 07:34:01.779597 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.779142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:40458)
0926 07:34:01.779300 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.779407 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:44905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:34:01.780172 29549 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 -> Peer 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124:44905): Couldn't send request to peer 34d9ed909226465db4db83e3a38d6f3b. Status: Network error: Client connection negotiation failed: client connection to 127.22.221.124:44905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:34:01.780648 30171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 LEARNER]: Refusing update from remote peer 02fa98fbcd424706b89819a21d1e0290: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 07:34:01.781297 30120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 07:34:01.781682 30313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:01.782030 30124 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:01.782358 30124 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.005s
I20260926 07:34:01.782457 30124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
uuid: "34d9ed909226465db4db83e3a38d6f3b"
format_stamp: "Formatted at 2026-09-26 07:34:01 on dist-test-slave-qlq8"
server_key: "c065037d2c9be4b2c150c722dc710f79"
server_key_iv: "405d9637e62d60ca30ac78b7791caafa"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.784907 30308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 940f3dedb10544cea8abc0ba623aef3e (127.22.221.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.787859 30124 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:01.790050 30171 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 940f3dedb10544cea8abc0ba623aef3e (127.22.221.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.794225 30199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.794471 30199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:01.794469 30124 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:34:01.794611 30124 log.cc:1196] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 07:34:01.795048 30124 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Deleting consensus metadata
I20260926 07:34:01.795421 30124 master_runner.cc:315] Copying system tablet from 127.22.221.126:42459
I20260926 07:34:01.795480 30120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.795739 30120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:01.796103 30120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.796332 30120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:01.798231 30124 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: tablet copy: Beginning tablet copy session from remote peer at address 127.22.221.126:42459
I20260926 07:34:01.817289 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.799030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.22.221.126:42459)
0926 07:34:01.799272 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:01.799277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:01.803153 (+ 3876us) server_negotiation.cc:407] Connection header received
0926 07:34:01.803378 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:01.803386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:01.803500 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:01.803606 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:01.804765 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.805939 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.807777 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.808411 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.808557 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.810785 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:01.810792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:01.810797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:01.810834 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:01.814268 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.815047 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.815053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.815058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.815278 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:01.815718 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:01.815723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:01.815727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:01.816080 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:01.816234 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:01.816679 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 07:34:01.816897 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 07:34:01.818379 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:01.798945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:48912)
0926 07:34:01.803027 (+ 4082us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:01.803047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:01.803195 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:01.803656 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:01.803664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:01.804339 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:01.804601 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.804614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.806107 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.806113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.807631 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:01.807645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:01.808596 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:01.808602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:01.808779 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:01.809520 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:01.809542 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:01.810575 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:01.814413 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.814418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.814422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.814910 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.815392 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:01.815397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:01.815400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:01.815592 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:01.816214 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:01.816219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:01.816414 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 07:34:01.816561 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:34:01.816888 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3920,"thread_start_us":3813,"threads_started":1}
I20260926 07:34:01.819041 29583 tablet_copy_service.cc:140] P 02fa98fbcd424706b89819a21d1e0290: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 34d9ed909226465db4db83e3a38d6f3b ({username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:48912)
I20260926 07:34:01.819203 29583 tablet_copy_service.cc:161] P 02fa98fbcd424706b89819a21d1e0290: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 34d9ed909226465db4db83e3a38d6f3b at {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:48912: session id = 34d9ed909226465db4db83e3a38d6f3b-00000000000000000000000000000000
I20260926 07:34:01.822849 29583 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:34:01.824817 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:34:01.828707 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29624
I20260926 07:34:01.831907 30124 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: tablet copy: Starting download of 0 data blocks...
I20260926 07:34:01.832202 30124 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: tablet copy: Starting download of 1 WAL segments...
I20260926 07:34:01.834602 30124 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:34:01.839663 29583 tablet_copy_service.cc:342] P 02fa98fbcd424706b89819a21d1e0290: Request end of tablet copy session 34d9ed909226465db4db83e3a38d6f3b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:48912
I20260926 07:34:01.839815 29583 tablet_copy_service.cc:433] P 02fa98fbcd424706b89819a21d1e0290: ending tablet copy session 34d9ed909226465db4db83e3a38d6f3b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 34d9ed909226465db4db83e3a38d6f3b
I20260926 07:34:01.851156 30124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:34:01.851590 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:40833
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:42459
--webserver_port=40825
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:01.854117 30320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:01.856972 30321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:01.870745 30124 server_base.cc:1034] running on GCE node
W20260926 07:34:01.871765 30324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:01.872243 30124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:01.873034 30124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:01.874394 30124 hybrid_clock.cc:648] HybridClock initialized: now 1790408041874340 us; error 48 us; skew 500 ppm
I20260926 07:34:01.874773 30124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:01 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408041, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:01.877244 30124 init.cc:377] Logged in from keytab as oryx/127.22.221.124@KRBTEST.COM (short username oryx)
I20260926 07:34:01.878592 30124 webserver.cc:460] Webserver started at http://127.22.221.124:45823/ using document root <none> and password file <none>
I20260926 07:34:01.878973 30124 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:01.879081 30124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:01.903685 30124 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.006s sys 0.016s
I20260926 07:34:01.905740 30331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:01.906255 30124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 07:34:01.906381 30124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
uuid: "34d9ed909226465db4db83e3a38d6f3b"
format_stamp: "Formatted at 2026-09-26 07:34:01 on dist-test-slave-qlq8"
server_key: "c065037d2c9be4b2c150c722dc710f79"
server_key_iv: "405d9637e62d60ca30ac78b7791caafa"
server_key_version: "encryptionkey@0"
I20260926 07:34:01.906567 30124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:01.925184 30124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:01.929215 30124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:01.956260 30124 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.124:44905
I20260926 07:34:01.957334 30124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
I20260926 07:34:01.956316 30382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.124:44905 every 8 connection(s)
I20260926 07:34:01.965600 30383 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:34:01.971648 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Bootstrap starting.
I20260926 07:34:01.986649 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:34:01.987112 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Bootstrap complete.
I20260926 07:34:01.988016 30383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } }
I20260926 07:34:01.988229 30383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34d9ed909226465db4db83e3a38d6f3b, State: Initialized, Role: LEARNER
I20260926 07:34:01.988489 30383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } }
I20260926 07:34:01.991417 30386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:01.991809 30383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:01.991837 30386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: This master's current role is: LEARNER
I20260926 07:34:02.000242 30397 catalog_manager.cc:1259] Loaded cluster ID: c259eddfa4434248a44bffc58578daae
I20260926 07:34:02.000336 30397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: loading cluster ID for follower catalog manager: success
I20260926 07:34:02.003741 30397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: acquiring CA information for follower catalog manager: success
I20260926 07:34:02.006104 30397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:34:02.213819 30323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:02.214120 30323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:02.214283 30323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:02.222129 30323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:02.222254 30323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:02.222419 30323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:02.225905 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.219430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.22.221.124:44905)
0926 07:34:02.220074 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.220079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.220095 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:02.220156 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.220159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.220225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.220323 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:02.220759 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.221721 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.224055 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.224894 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.225048 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.225187 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.225531 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.225662 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":348,"threads_started":1}
I20260926 07:34:02.225905 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.219211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:40460)
0926 07:34:02.219409 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.219430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.219554 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.220378 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.220384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.220428 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:02.220608 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.220619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.221861 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.221866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.223872 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.223890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.225079 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.225084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.225320 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.225359 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.225471 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:34:02.225623 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:34:02.227029 30358 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 07:34:02.227743 30386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fa98fbcd424706b89819a21d1e0290. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:02.228094 30386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: This master's current role is: LEARNER
I20260926 07:34:02.234369 30323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:40833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=40825
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:02.235610 30323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:02.236825 30323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:02.249588 30404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.249545 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.250142 30403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.251047 30323 server_base.cc:1034] running on GCE node
I20260926 07:34:02.251644 30323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:02.252530 30323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:02.253759 30323 hybrid_clock.cc:648] HybridClock initialized: now 1790408042253711 us; error 41 us; skew 500 ppm
I20260926 07:34:02.254057 30323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:02.257301 30323 init.cc:377] Logged in from keytab as oryx/127.22.221.65@KRBTEST.COM (short username oryx)
I20260926 07:34:02.258836 30323 webserver.cc:460] Webserver started at http://127.22.221.65:40825/ using document root <none> and password file <none>
I20260926 07:34:02.259550 30323 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:02.259703 30323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:02.264782 30323 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:34:02.267318 30413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.268636 30323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:02.268797 30323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "64229b9e03ab45018198fe4555158212"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "83b235f66b7d388414f9a6328b0fe09d"
server_key_iv: "7c1c62632058e451945a942d31a9848b"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.269744 30323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:02.288241 30323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:02.291885 30323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:02.292140 30323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:02.293141 30323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:02.294478 30323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:02.294589 30323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.294711 30323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:02.294804 30323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.327625 30323 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:40833
I20260926 07:34:02.327718 30525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:40833 every 8 connection(s)
I20260926 07:34:02.329025 30323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:02.329728 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 30323
I20260926 07:34:02.330108 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29758
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.66@KRBTEST.COM
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.124@KRBTEST.COM
I20260926 07:34:02.368269 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.331187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.22.221.66:35679)
0926 07:34:02.341865 (+ 10678us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.341877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.342878 (+ 1001us) server_negotiation.cc:407] Connection header received
0926 07:34:02.342980 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.342992 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.343305 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.343467 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.346082 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.347956 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.350413 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.351486 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.351616 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.360198 (+ 8582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.360213 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.360232 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.360298 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.364549 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.365712 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.365721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.365724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.365866 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.366331 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.366336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.366340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.366734 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.367004 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.367588 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.367840 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10461,"thread_start_us":10651,"threads_started":1}
I20260926 07:34:02.370366 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.333533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:34313 (local address 127.22.221.124:44905)
0926 07:34:02.333774 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.333782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.344164 (+ 10382us) server_negotiation.cc:407] Connection header received
0926 07:34:02.346024 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.346030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.346115 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.346209 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.348669 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.350143 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.353331 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.354192 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.354306 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.362777 (+ 8471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.362787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.362793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.362842 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.366558 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.367613 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.367619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.367621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.367765 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.368317 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.368322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.368325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.368667 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.368785 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.369828 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.370062 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:34:02.370710 29881 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:02.372558 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.332711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.22.221.65:34313)
0926 07:34:02.344005 (+ 11294us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.344044 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.345720 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.346591 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.346602 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.347398 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.348417 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.348449 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.350443 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.350451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.353159 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.353173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.354673 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.354682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.355049 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.355847 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.355898 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.362475 (+ 6577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.366712 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.366723 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.366740 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.367476 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.367927 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.367932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.367936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.368176 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.368882 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.368889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.369371 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.370624 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0926 07:34:02.371143 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11044,"thread_start_us":13203,"threads_started":1}
I20260926 07:34:02.374260 30526 heartbeater.cc:347] Connected to a master server at 127.22.221.124:44905
I20260926 07:34:02.374573 30526 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:02.375738 30526 heartbeater.cc:511] Master 127.22.221.124:44905 requested a full tablet report, sending...
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
I20260926 07:34:02.379271 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.337285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:53463)
0926 07:34:02.361611 (+ 24326us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.361631 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.361751 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.362407 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.362413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.362988 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.363347 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.363360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.365244 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.365251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.366855 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.366872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.368211 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.368219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.368400 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.368963 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.368983 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.371883 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.376173 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.376180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.376187 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.376748 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.377199 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.377203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.377206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.377360 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.378061 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.378066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.378208 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.378337 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 07:34:02.378616 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24186,"thread_start_us":30593,"threads_started":1}
I20260926 07:34:02.379676 30348 ts_manager.cc:194] Registered new tserver with Master: 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:02.380265 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.337822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:53463 (local address 127.22.221.126:42459)
0926 07:34:02.361110 (+ 23288us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.361121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.361866 (+ 745us) server_negotiation.cc:407] Connection header received
0926 07:34:02.361935 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.361940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.362019 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.362125 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.363519 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.364710 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.367025 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.367662 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.367780 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.372206 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.372218 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.372226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.372279 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.376022 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.376899 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.376904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.376907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.377065 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.377504 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.377509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.377512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.377834 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.378000 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.379184 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.379512 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23137,"thread_start_us":23301,"threads_started":1}
I20260926 07:34:02.381138 30527 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42459
I20260926 07:34:02.381307 30527 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:02.381825 30527 heartbeater.cc:511] Master 127.22.221.126:42459 requested a full tablet report, sending...
I20260926 07:34:02.383033 29563 ts_manager.cc:194] Re-registered known tserver with Master: 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:02.385402 29563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:53463
I20260926 07:34:02.387984 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.353205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.22.221.65:46273)
0926 07:34:02.365459 (+ 12254us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.365479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.365615 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.366186 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.366190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.366795 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.367494 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.367506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.369201 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.369205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.371176 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.371193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.372305 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.372311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.372505 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.374206 (+ 1701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.374236 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.377635 (+ 3399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.385257 (+ 7622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.385265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.385270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.385705 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.386115 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.386120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.386124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.386305 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.386923 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.386927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.387167 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.387330 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 07:34:02.387620 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12100,"mutex_wait_us":39,"thread_start_us":13771,"threads_started":1}
I20260926 07:34:02.388083 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.347100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:46273 (local address 127.22.221.125:44963)
0926 07:34:02.365161 (+ 18061us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.365167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.365724 (+ 557us) server_negotiation.cc:407] Connection header received
0926 07:34:02.365792 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.365797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.365886 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.365988 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.367659 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.369045 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.371358 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.371965 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.372080 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.377896 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.377905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.377913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.377965 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.385116 (+ 7151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.385818 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.385824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.385828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.385970 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.386460 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.386465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.386469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.386791 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.386940 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.387462 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.387684 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17928,"thread_start_us":17824,"threads_started":1}
I20260926 07:34:02.388921 30528 heartbeater.cc:347] Connected to a master server at 127.22.221.125:44963
I20260926 07:34:02.389063 30528 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:02.389492 30528 heartbeater.cc:511] Master 127.22.221.125:44963 requested a full tablet report, sending...
I20260926 07:34:02.391391 30161 ts_manager.cc:194] Registered new tserver with Master: 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:02.466933 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29758
I20260926 07:34:02.485316 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:35679
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:42459
--webserver_port=37105
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:02.689685 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: NON_VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: true } } }
I20260926 07:34:02.690171 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:02.691857 30539 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: attempting to promote NON_VOTER 34d9ed909226465db4db83e3a38d6f3b to VOTER
I20260926 07:34:02.692436 30539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } }
I20260926 07:34:02.694305 30171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Refusing update from remote peer 02fa98fbcd424706b89819a21d1e0290: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 07:34:02.694458 30358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 LEARNER]: Refusing update from remote peer 02fa98fbcd424706b89819a21d1e0290: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 07:34:02.695034 30539 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 07:34:02.695464 30540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 07:34:02.698515 30539 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.699064 30171 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.702497 30539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.702759 30539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:02.703301 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.703575 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:02.704437 30358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.710572 30386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.710841 30386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:02.712669 30543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.713155 30543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:02.713333 30544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02fa98fbcd424706b89819a21d1e0290" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:02.713582 30544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290 [sys.catalog]: This master's current role is: LEADER
W20260926 07:34:02.808805 30538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:02.809144 30538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:02.809322 30538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:02.817348 30538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:02.817497 30538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:02.817673 30538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:02.827585 30538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:35679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=37105
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:02.828857 30538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:02.830164 30538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:02.843302 30552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.843391 30550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.843389 30549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.849216 30538 server_base.cc:1034] running on GCE node
I20260926 07:34:02.849848 30538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:02.850782 30538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:02.852016 30538 hybrid_clock.cc:648] HybridClock initialized: now 1790408042851988 us; error 41 us; skew 500 ppm
I20260926 07:34:02.852340 30538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:02.855509 30538 init.cc:377] Logged in from keytab as oryx/127.22.221.66@KRBTEST.COM (short username oryx)
I20260926 07:34:02.856848 30538 webserver.cc:460] Webserver started at http://127.22.221.66:37105/ using document root <none> and password file <none>
I20260926 07:34:02.857475 30538 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:02.857590 30538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:02.862854 30538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:02.865454 30559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.866726 30538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:34:02.866855 30538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "09c9aa2ed60d443c99b34d681bad4294"
format_stamp: "Formatted at 2026-09-26 07:33:59 on dist-test-slave-qlq8"
server_key: "d87e69f049e0f617624b31aa17023585"
server_key_iv: "72a8954b0b43ca5d6fdc8137c413e38f"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.867827 30538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:02.888689 30538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:02.892740 30538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:02.892984 30538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:02.893824 30538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:02.895272 30538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:02.895370 30538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.895495 30538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:02.895555 30538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.929636 30538 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:35679
I20260926 07:34:02.929725 30671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:35679 every 8 connection(s)
I20260926 07:34:02.931054 30538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:02.935508 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 30538
I20260926 07:34:02.935840 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29891
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.124@KRBTEST.COM
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
Sep 26 07:34:02 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.67@KRBTEST.COM
I20260926 07:34:03.000449 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.960025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:34969 (local address 127.22.221.125:44963)
0926 07:34:02.971889 (+ 11864us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.971897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.973160 (+ 1263us) server_negotiation.cc:407] Connection header received
0926 07:34:02.974426 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.974434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.974538 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.974649 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.977103 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.978640 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.982996 (+ 4356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.983636 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.983757 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.993637 (+ 9880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.993644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.993648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.993708 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.997174 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.998064 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.998071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.998077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.998251 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.998718 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.998722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.998725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.999096 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.999269 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.999862 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.000077 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11708,"thread_start_us":11872,"threads_started":1}
I20260926 07:34:03.001336 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.934998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.22.221.66:60399)
0926 07:34:02.957024 (+ 22026us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.957094 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.973810 (+ 16716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.981506 (+ 7696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.981511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.982032 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.982405 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.982422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.984191 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.984199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.985729 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.985742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.986807 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.986813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.987040 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.987815 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.987846 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.991183 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.995045 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.995057 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.995074 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.995686 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.997565 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.997570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.997574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.997832 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.998574 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.998582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.999029 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.999354 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 07:34:02.999910 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21673,"mutex_wait_us":34,"thread_start_us":23668,"threads_started":1}
I20260926 07:34:03.001992 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.934285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:51669 (local address 127.22.221.126:42459)
0926 07:34:02.981812 (+ 47527us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.981818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.981835 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:34:02.981891 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.981896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.981971 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.985011 (+ 3040us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.985173 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.986223 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.988761 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.989460 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.989602 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.993595 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.993602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.993608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.993649 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.997693 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.998578 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.998585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.998591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.998750 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.999258 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.999264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.999267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.999591 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.999749 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.001391 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.001633 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47406,"thread_start_us":46427,"threads_started":1}
I20260926 07:34:03.002434 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.934342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:60399 (local address 127.22.221.124:44905)
0926 07:34:02.981183 (+ 46841us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.981189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.981206 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:34:02.981253 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.981258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.981323 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.981461 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.982690 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.983968 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.985894 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.986596 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.986724 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.991470 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.991476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.991481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.991525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.994877 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.995808 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.995815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.995819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.997412 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.997975 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.997980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.997984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.998349 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.998483 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.001822 (+ 3339us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.002043 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46665,"thread_start_us":46526,"threads_started":1}
I20260926 07:34:03.002435 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.969460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.22.221.66:34969)
0926 07:34:02.972977 (+ 3517us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.973005 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.974262 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.975106 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.975118 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.976060 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.976844 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.976869 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.978945 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.978950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.982819 (+ 3869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.982833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.986462 (+ 3629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.986469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.986747 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.987974 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.987996 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.992587 (+ 4591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.997313 (+ 4726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.997321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.997326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.997849 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.998426 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.998430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.998433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.998593 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.999330 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.999334 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.999469 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.999610 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.001070 (+ 1460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2786,"mutex_wait_us":1750,"thread_start_us":9262,"threads_started":1}
I20260926 07:34:03.003139 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.934998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:51669)
0926 07:34:02.968507 (+ 33509us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.968540 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.973970 (+ 5430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.983974 (+ 10004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.983978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.984529 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.984870 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.984880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.986967 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.986975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.988557 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.988595 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.989669 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.989677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.989900 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.990448 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.990471 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.993376 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.997856 (+ 4480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.997861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.997867 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.998433 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.998891 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.998895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.998898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.999095 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.999819 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.999825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.999972 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.000123 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.000395 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33302,"mutex_wait_us":228,"thread_start_us":33437,"threads_started":1}
I20260926 07:34:03.003469 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.937331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.22.221.67:32773)
0926 07:34:02.978860 (+ 41529us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.978870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.981024 (+ 2154us) server_negotiation.cc:407] Connection header received
0926 07:34:02.981102 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.981110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.981384 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.981564 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.983813 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.985408 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.987313 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.988700 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.988826 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.996218 (+ 7392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.996232 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.996250 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.996304 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.000251 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.001061 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.001069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.001073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.001236 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.001686 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.001691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.001694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.002083 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.002478 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.002794 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.003045 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41339,"thread_start_us":41527,"threads_started":1}
I20260926 07:34:03.004294 30674 heartbeater.cc:347] Connected to a master server at 127.22.221.125:44963
I20260926 07:34:03.004388 30672 heartbeater.cc:347] Connected to a master server at 127.22.221.124:44905
I20260926 07:34:03.004616 30674 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.004623 30672 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.004791 30014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:03.005203 30673 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42459
I20260926 07:34:03.005347 30673 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.005788 30674 heartbeater.cc:511] Master 127.22.221.125:44963 requested a full tablet report, sending...
I20260926 07:34:03.005825 30672 heartbeater.cc:511] Master 127.22.221.124:44905 requested a full tablet report, sending...
I20260926 07:34:03.005875 30673 heartbeater.cc:511] Master 127.22.221.126:42459 requested a full tablet report, sending...
I20260926 07:34:03.007721 30348 ts_manager.cc:194] Registered new tserver with Master: 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679)
I20260926 07:34:03.007741 30161 ts_manager.cc:194] Registered new tserver with Master: 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679)
I20260926 07:34:03.007717 29563 ts_manager.cc:194] Re-registered known tserver with Master: 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679)
I20260926 07:34:03.009262 29563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:51669
I20260926 07:34:03.094938 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29891
I20260926 07:34:03.110874 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:32773
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:42459
--webserver_port=41417
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:03.388095 30527 heartbeater.cc:503] Master 127.22.221.126:42459 was elected leader, sending a full tablet report...
W20260926 07:34:03.434437 30685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:03.434695 30685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:03.434787 30685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:03.441955 30685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:03.442051 30685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:03.442160 30685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:03.451521 30685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:32773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=41417
--tserver_master_addrs=127.22.221.126:42459,127.22.221.125:44963,127.22.221.124:44905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:03.452840 30685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:03.454165 30685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:03.467427 30690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.467409 30691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.467478 30693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:03.468403 30685 server_base.cc:1034] running on GCE node
I20260926 07:34:03.468964 30685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:03.469839 30685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:03.471066 30685 hybrid_clock.cc:648] HybridClock initialized: now 1790408043471010 us; error 61 us; skew 500 ppm
I20260926 07:34:03.471448 30685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:03 dist-test-slave-qlq8 krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408043, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:03.474691 30685 init.cc:377] Logged in from keytab as oryx/127.22.221.67@KRBTEST.COM (short username oryx)
I20260926 07:34:03.476052 30685 webserver.cc:460] Webserver started at http://127.22.221.67:41417/ using document root <none> and password file <none>
I20260926 07:34:03.476698 30685 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:03.476836 30685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:03.482213 30685 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:34:03.484746 30700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.485944 30685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:34:03.486102 30685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "38384ae8c9ad4ebc8b0e04c128a56039"
format_stamp: "Formatted at 2026-09-26 07:34:00 on dist-test-slave-qlq8"
server_key: "299ed11665ace6c13069421c4114f205"
server_key_iv: "190323c6a30aca6f7f987ea2c8ea5e3f"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.487026 30685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:03.503448 30685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:03.507296 30685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:03.507512 30685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:03.508379 30685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:03.509536 30685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:03.509629 30685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.509706 30685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:03.509760 30685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.541273 30685 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:32773
I20260926 07:34:03.541356 30812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:32773 every 8 connection(s)
I20260926 07:34:03.542701 30685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:03.550484 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 30685
Sep 26 07:34:03 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408043, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
Sep 26 07:34:03 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408043, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.124@KRBTEST.COM
Sep 26 07:34:03 dist-test-slave-qlq8 krb5kdc[29518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408043, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.125@KRBTEST.COM
I20260926 07:34:03.598842 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.568542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:44755 (local address 127.22.221.124:44905)
0926 07:34:03.569180 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.569190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.575698 (+ 6508us) server_negotiation.cc:407] Connection header received
0926 07:34:03.575909 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.575913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.575972 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.576082 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.577409 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.578583 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.581600 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.582232 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.582335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.588292 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.588302 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.588309 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.588360 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.592376 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.593634 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.593643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.593649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.593829 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.594539 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.594547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.594553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.595042 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.595244 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.597987 (+ 2743us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.598305 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":323,"threads_started":1}
I20260926 07:34:03.601796 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.569124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.22.221.67:44755)
0926 07:34:03.575588 (+ 6464us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.575606 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.575762 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.576115 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.576119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.576719 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:03.577165 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.577182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.578803 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.578808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.581425 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.581438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.582619 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.582632 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.582844 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.583541 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:03.583584 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:03.588041 (+ 4457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:03.592540 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.592549 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.592564 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.593459 (+ 895us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.593979 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.593987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.593993 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.594347 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.595333 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:03.595392 (+ 59us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:03.596025 (+ 633us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.596402 (+ 377us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.600013 (+ 3611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6335,"mutex_wait_us":875,"thread_start_us":6210,"threads_started":1}
I20260926 07:34:03.604534 30813 heartbeater.cc:347] Connected to a master server at 127.22.221.124:44905
I20260926 07:34:03.604882 30813 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.605846 30813 heartbeater.cc:511] Master 127.22.221.124:44905 requested a full tablet report, sending...
I20260926 07:34:03.606865 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.546120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:42279)
0926 07:34:03.568145 (+ 22025us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.568169 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.568300 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.570661 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.570677 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.571816 (+ 1139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:03.572885 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.572913 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.575616 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.575624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.578418 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.578438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.580291 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.580299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.580657 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.581501 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:03.581536 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:03.584803 (+ 3267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:03.596947 (+ 12144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.596955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.596961 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.597665 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.603545 (+ 5880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.603551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.603557 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.603703 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.605515 (+ 1812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:03.605520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:03.605699 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.605864 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.606213 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21823,"mutex_wait_us":104,"spinlock_wait_cycles":20736,"thread_start_us":17132,"threads_started":1}
I20260926 07:34:03.608373 30348 ts_manager.cc:194] Registered new tserver with Master: 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773)
I20260926 07:34:03.610044 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.546120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.22.221.67:32867)
0926 07:34:03.563850 (+ 17730us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.563895 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.567193 (+ 3298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.576621 (+ 9428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.576626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.577086 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:03.577401 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.577412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.579396 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.579401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.580863 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.580882 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.589127 (+ 8245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.589150 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.589366 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.589898 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:03.589922 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:03.592583 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:03.602268 (+ 9685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.602273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.602278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.602735 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.603331 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.603337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.603340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.603525 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.608991 (+ 5466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:03.608997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:03.609170 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.609316 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.609643 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17436,"mutex_wait_us":1,"thread_start_us":21544,"threads_started":1}
I20260926 07:34:03.611402 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.545550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:32867 (local address 127.22.221.125:44963)
0926 07:34:03.575345 (+ 29795us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.575354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.575377 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:34:03.575462 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.575470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.576271 (+ 801us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.576392 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.577590 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.578649 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.588003 (+ 9354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.588925 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.589091 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.592777 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.592787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.592794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.592874 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.602117 (+ 9243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.602986 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.602991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.602994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.603173 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.604039 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.604043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.604046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.604434 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.609058 (+ 4624us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.610432 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.610799 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29617,"thread_start_us":29508,"threads_started":1}
I20260926 07:34:03.610090 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.545817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:42279 (local address 127.22.221.126:42459)
0926 07:34:03.569934 (+ 24117us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.569942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.569968 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:34:03.570044 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.570052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.570153 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.573119 (+ 2966us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.573328 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.574764 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.578792 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.580025 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.580387 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.586428 (+ 6041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.586438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.586446 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.586500 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.596805 (+ 10305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.602988 (+ 6183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.602996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.603003 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.603197 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.604873 (+ 1676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.604881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.604887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.605274 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.605411 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.609263 (+ 3852us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.609580 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23959,"thread_start_us":23843,"threads_started":1}
I20260926 07:34:03.612636 30814 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42459
I20260926 07:34:03.612774 30814 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.612803 30815 heartbeater.cc:347] Connected to a master server at 127.22.221.125:44963
I20260926 07:34:03.612965 30815 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.613363 30814 heartbeater.cc:511] Master 127.22.221.126:42459 requested a full tablet report, sending...
I20260926 07:34:03.613462 30815 heartbeater.cc:511] Master 127.22.221.125:44963 requested a full tablet report, sending...
I20260926 07:34:03.614348 29563 ts_manager.cc:194] Re-registered known tserver with Master: 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773)
I20260926 07:34:03.614573 30161 ts_manager.cc:194] Registered new tserver with Master: 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773)
I20260926 07:34:03.615803 29563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:42279
I20260926 07:34:03.625734 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.581198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.22.221.126:42459)
0926 07:34:03.602656 (+ 21458us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.602664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.604418 (+ 1754us) server_negotiation.cc:407] Connection header received
0926 07:34:03.604473 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.604477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.604569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.604669 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.608500 (+ 3831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.611149 (+ 2649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.613868 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.614529 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.614652 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.618610 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.618617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.618621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.618654 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.622544 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.623503 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.623510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.623517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.623671 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.624117 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.624123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.624127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.624455 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.624727 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.625080 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.625307 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21314,"thread_start_us":21632,"threads_started":1}
I20260926 07:34:03.628203 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.583076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.22.221.125:44963)
0926 07:34:03.602287 (+ 19211us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.602292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.605598 (+ 3306us) server_negotiation.cc:407] Connection header received
0926 07:34:03.605655 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.605659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.605730 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.605829 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.610438 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.612522 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.615551 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.616342 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.617735 (+ 1393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.621120 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.621128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.621134 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.621172 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.625037 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.625898 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.625904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.625907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.626075 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.626493 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.626498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.626501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.626867 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.627118 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.627564 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.627787 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":7285,"thread_start_us":6502,"threads_started":1}
I20260926 07:34:03.631392 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.605515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.22.221.124:44905)
0926 07:34:03.605684 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.605689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.614931 (+ 9242us) server_negotiation.cc:407] Connection header received
0926 07:34:03.615189 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.615194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.615293 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.615385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.617741 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.618713 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.621263 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.621657 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.621800 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.624863 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.624869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.624873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.624909 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.628846 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.629661 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.629665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.629667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.629784 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.630160 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.630163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.630165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.630430 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.630600 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.630939 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.631099 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:03.630069 29563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:03.637603 29563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:03.681150 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.655376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.22.221.66:35679)
0926 07:34:03.671838 (+ 16462us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.671850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.671887 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:34:03.672019 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.672031 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.672438 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.672697 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.673390 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.675456 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.678622 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.680069 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.680205 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.680413 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.680690 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.680868 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16198,"thread_start_us":16017,"threads_started":1}
I20260926 07:34:03.681687 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.655108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:35679 (local address 127.0.0.1:47180)
0926 07:34:03.662110 (+ 7002us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.662138 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.662290 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.672703 (+ 10413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.672711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.672749 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.673195 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.673212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.675676 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.675685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.678451 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.678464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.680280 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.680288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.680475 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.680512 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.680626 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.680854 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":62,"thread_start_us":322,"threads_started":1}
I20260926 07:34:03.684707 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.655824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:32773 (local address 127.0.0.1:52962)
0926 07:34:03.676221 (+ 20397us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.676238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.676375 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.677188 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.677193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.677226 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.677594 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.677605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.679423 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.679429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.682124 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.682136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.683867 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.683873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.684019 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.684061 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.684142 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.684258 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20270,"mutex_wait_us":25,"thread_start_us":21599,"threads_started":1}
I20260926 07:34:03.684717 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.656334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.22.221.67:32773)
0926 07:34:03.668573 (+ 12239us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.668579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.676327 (+ 7748us) server_negotiation.cc:407] Connection header received
0926 07:34:03.676547 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.676554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.676788 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.677202 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.677778 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.679278 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.682252 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.683520 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.683647 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.684170 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.684251 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.684412 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12077,"thread_start_us":11905,"threads_started":1}
I20260926 07:34:03.685590 30607 tablet_service.cc:1467] Processing CreateTablet for tablet 50355f585e77484da8f9d0926d34a721 (DEFAULT_TABLE table=test-table [id=b0902d2ef50e4fb8b87f90e30deb0a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:03.686578 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.655834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40833 (local address 127.0.0.1:54498)
0926 07:34:03.677069 (+ 21235us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.677088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.677202 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.677927 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.677933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.677957 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.678381 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.678393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.680075 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.680082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.683675 (+ 3593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.683689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.685461 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.685467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.685649 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.685681 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.686077 (+ 396us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.686193 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21093,"mutex_wait_us":43,"thread_start_us":20120,"threads_started":1}
I20260926 07:34:03.687050 30607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50355f585e77484da8f9d0926d34a721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.687304 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.656293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.22.221.65:40833)
0926 07:34:03.675701 (+ 19408us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.675713 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.677332 (+ 1619us) server_negotiation.cc:407] Connection header received
0926 07:34:03.677435 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.677442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.677690 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.677935 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.678572 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.679956 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.683837 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.685251 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.685377 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.685568 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.686788 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.686953 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19206,"thread_start_us":19085,"threads_started":1}
I20260926 07:34:03.687849 30748 tablet_service.cc:1467] Processing CreateTablet for tablet 50355f585e77484da8f9d0926d34a721 (DEFAULT_TABLE table=test-table [id=b0902d2ef50e4fb8b87f90e30deb0a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:03.688818 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50355f585e77484da8f9d0926d34a721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.693914 30461 tablet_service.cc:1467] Processing CreateTablet for tablet 50355f585e77484da8f9d0926d34a721 (DEFAULT_TABLE table=test-table [id=b0902d2ef50e4fb8b87f90e30deb0a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:03.695196 30461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50355f585e77484da8f9d0926d34a721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.702176 30846 tablet_bootstrap.cc:492] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Bootstrap starting.
I20260926 07:34:03.705035 30846 tablet_bootstrap.cc:654] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.705986 30846 log.cc:824] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.707307 30847 tablet_bootstrap.cc:492] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Bootstrap starting.
I20260926 07:34:03.709156 30846 tablet_bootstrap.cc:492] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: No bootstrap required, opened a new log
I20260926 07:34:03.709429 30846 ts_tablet_manager.cc:1397] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 07:34:03.710904 30849 tablet_bootstrap.cc:492] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Bootstrap starting.
I20260926 07:34:03.711593 30847 tablet_bootstrap.cc:654] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.712771 30847 log.cc:824] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.713753 30849 tablet_bootstrap.cc:654] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.714669 30849 log.cc:824] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.715616 30847 tablet_bootstrap.cc:492] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: No bootstrap required, opened a new log
I20260926 07:34:03.715875 30847 ts_tablet_manager.cc:1397] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 07:34:03.717687 30849 tablet_bootstrap.cc:492] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: No bootstrap required, opened a new log
I20260926 07:34:03.717922 30849 ts_tablet_manager.cc:1397] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 07:34:03.719674 30846 raft_consensus.cc:348] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.720101 30846 raft_consensus.cc:374] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.720206 30846 raft_consensus.cc:729] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38384ae8c9ad4ebc8b0e04c128a56039, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.720806 30846 consensus_queue.cc:260] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.721880 30814 heartbeater.cc:503] Master 127.22.221.126:42459 was elected leader, sending a full tablet report...
I20260926 07:34:03.723987 30846 ts_tablet_manager.cc:1428] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 07:34:03.728325 30847 raft_consensus.cc:348] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.728665 30847 raft_consensus.cc:374] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.728765 30847 raft_consensus.cc:729] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9aa2ed60d443c99b34d681bad4294, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.729295 30847 consensus_queue.cc:260] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.729915 30849 raft_consensus.cc:348] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.730274 30849 raft_consensus.cc:374] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.730299 30673 heartbeater.cc:503] Master 127.22.221.126:42459 was elected leader, sending a full tablet report...
I20260926 07:34:03.730362 30849 raft_consensus.cc:729] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64229b9e03ab45018198fe4555158212, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.730924 30847 ts_tablet_manager.cc:1428] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260926 07:34:03.730893 30849 consensus_queue.cc:260] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.733233 30849 ts_tablet_manager.cc:1428] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260926 07:34:03.772185 30852 raft_consensus.cc:482] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:03.772516 30852 raft_consensus.cc:504] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.774400 30852 leader_election.cc:288] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679), 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:03.781005 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.774806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:35679 (local address 127.22.221.67:51579)
0926 07:34:03.775200 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.775221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.775382 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.775831 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.775838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.775879 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.776074 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.776085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.777337 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.777344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.778861 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.778872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.780266 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.780272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.780459 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.780483 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.780594 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.780722 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260926 07:34:03.781001 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.774801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:51579 (local address 127.22.221.66:35679)
0926 07:34:03.775041 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.775048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.775316 (+ 268us) server_negotiation.cc:407] Connection header received
0926 07:34:03.775543 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.775552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.775635 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.775744 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.776243 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.777206 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.779017 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.780080 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.780241 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.780314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.780621 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.780744 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:34:03.781438 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.775682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:33161 (local address 127.22.221.65:40833)
0926 07:34:03.775899 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.775905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.775925 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:03.775991 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.775996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.776089 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.776183 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.776717 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.777587 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.779490 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.780496 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.780673 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.780746 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.781055 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.781191 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:03.781437 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.775286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40833 (local address 127.22.221.67:33161)
0926 07:34:03.775550 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.775569 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.775657 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.776368 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.776375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.776404 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.776550 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.776558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.777736 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.777742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.779347 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.779367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.780661 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.780668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.780875 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.780899 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.781027 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.781178 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 07:34:03.782018 30627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50355f585e77484da8f9d0926d34a721" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9aa2ed60d443c99b34d681bad4294" is_pre_election: true
I20260926 07:34:03.782500 30627 raft_consensus.cc:2454] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 0.
I20260926 07:34:03.782397 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50355f585e77484da8f9d0926d34a721" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64229b9e03ab45018198fe4555158212" is_pre_election: true
I20260926 07:34:03.782900 30481 raft_consensus.cc:2454] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 0.
I20260926 07:34:03.783290 30704 leader_election.cc:302] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9aa2ed60d443c99b34d681bad4294, 38384ae8c9ad4ebc8b0e04c128a56039; no voters:
I20260926 07:34:03.783695 30852 raft_consensus.cc:2781] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:03.783851 30852 raft_consensus.cc:482] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:03.783972 30852 raft_consensus.cc:3037] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.786135 30852 raft_consensus.cc:504] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.786675 30852 leader_election.cc:288] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 election: Requested vote from peers 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679), 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:03.787166 30627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50355f585e77484da8f9d0926d34a721" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9aa2ed60d443c99b34d681bad4294"
I20260926 07:34:03.787328 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50355f585e77484da8f9d0926d34a721" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64229b9e03ab45018198fe4555158212"
I20260926 07:34:03.787421 30627 raft_consensus.cc:3037] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.787566 30481 raft_consensus.cc:3037] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.789600 30627 raft_consensus.cc:2454] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 1.
I20260926 07:34:03.789885 30481 raft_consensus.cc:2454] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 1.
I20260926 07:34:03.790097 30704 leader_election.cc:302] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9aa2ed60d443c99b34d681bad4294, 38384ae8c9ad4ebc8b0e04c128a56039; no voters:
I20260926 07:34:03.790398 30852 raft_consensus.cc:2781] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:03.790882 30852 raft_consensus.cc:686] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 LEADER]: Becoming Leader. State: Replica: 38384ae8c9ad4ebc8b0e04c128a56039, State: Running, Role: LEADER
I20260926 07:34:03.791522 30852 consensus_queue.cc:237] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } }
I20260926 07:34:03.796518 29563 catalog_manager.cc:5495] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } health_report { overall_health: HEALTHY } } }
W20260926 07:34:03.811179 30816 tablet.cc:2369] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:03.842005 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.836691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.22.221.67:32773)
0926 07:34:03.836896 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.836902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.836921 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:03.837124 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.837128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.837203 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.837354 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:03.837725 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.838431 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.839952 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.840351 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.840479 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.840958 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:03.841437 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:03.841618 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.841673 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.841749 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 07:34:03.844326 30532 tablet.cc:2369] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:03.850034 30627 raft_consensus.cc:1261] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Refusing update from remote peer 38384ae8c9ad4ebc8b0e04c128a56039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:03.850034 30481 raft_consensus.cc:1261] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Refusing update from remote peer 38384ae8c9ad4ebc8b0e04c128a56039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:03.850957 30855 consensus_queue.cc:1035] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:03.851368 30852 consensus_queue.cc:1035] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:03.863543 30860 mvcc.cc:205] Tried to move back new op lower bound from 7333511347599732736 to 7333511347374002176. Current Snapshot: MvccSnapshot[applied={T|T < 7333511347599732736}]
I20260926 07:34:03.880431 29562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:03.880745 29562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:03.889086 29562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02fa98fbcd424706b89819a21d1e0290: Sending DeleteTablet for 3 replicas of tablet 50355f585e77484da8f9d0926d34a721
I20260926 07:34:03.890836 30461 tablet_service.cc:1514] Processing DeleteTablet for tablet 50355f585e77484da8f9d0926d34a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:54498
I20260926 07:34:03.891464 30460 tablet_service.cc:1514] Processing DeleteTablet for tablet 50355f585e77484da8f9d0926d34a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:54498
I20260926 07:34:03.891981 30607 tablet_service.cc:1514] Processing DeleteTablet for tablet 50355f585e77484da8f9d0926d34a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:47180
I20260926 07:34:03.893926 30748 tablet_service.cc:1514] Processing DeleteTablet for tablet 50355f585e77484da8f9d0926d34a721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:52962
I20260926 07:34:03.899142 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 29531
I20260926 07:34:03.900292 30868 tablet_replica.cc:330] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: stopping tablet replica
I20260926 07:34:03.900341 29593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:03.900840 30868 raft_consensus.cc:2229] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:03.901360 30868 raft_consensus.cc:2258] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:03.904845 30871 tablet_replica.cc:330] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: stopping tablet replica
I20260926 07:34:03.905400 30871 raft_consensus.cc:2229] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:03.906325 30870 tablet_replica.cc:330] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: stopping tablet replica
I20260926 07:34:03.906666 30870 raft_consensus.cc:2229] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:03.907112 30871 raft_consensus.cc:2258] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:03.907202 30870 raft_consensus.cc:2258] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:03.909430 30870 ts_tablet_manager.cc:1905] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:03.909695 30868 ts_tablet_manager.cc:1905] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:03.916381 30870 ts_tablet_manager.cc:1918] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:03.916543 30870 log.cc:1196] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/50355f585e77484da8f9d0926d34a721
I20260926 07:34:03.917116 30870 ts_tablet_manager.cc:1939] T 50355f585e77484da8f9d0926d34a721 P 09c9aa2ed60d443c99b34d681bad4294: Deleting consensus metadata
I20260926 07:34:03.917448 30871 ts_tablet_manager.cc:1905] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:03.921168 30868 ts_tablet_manager.cc:1918] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:03.921339 30868 log.cc:1196] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/50355f585e77484da8f9d0926d34a721
I20260926 07:34:03.921900 30868 ts_tablet_manager.cc:1939] T 50355f585e77484da8f9d0926d34a721 P 64229b9e03ab45018198fe4555158212: Deleting consensus metadata
I20260926 07:34:03.925403 30871 ts_tablet_manager.cc:1918] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:03.925571 30871 log.cc:1196] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/50355f585e77484da8f9d0926d34a721
I20260926 07:34:03.926110 30871 ts_tablet_manager.cc:1939] T 50355f585e77484da8f9d0926d34a721 P 38384ae8c9ad4ebc8b0e04c128a56039: Deleting consensus metadata
I20260926 07:34:03.927497 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.892330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.22.221.125:44963)
0926 07:34:03.892770 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.892777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.892795 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:03.893732 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.893741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.893825 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.893899 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.896905 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.898383 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.910399 (+ 12016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.911041 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.911153 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.918273 (+ 7120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.918280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.918284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.918330 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.921867 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.924951 (+ 3084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.924957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.924961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.925124 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.925602 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.925607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.925611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.926002 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.926257 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.926848 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.927105 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260926 07:34:03.945487 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.893060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.22.221.124:44905)
0926 07:34:03.893218 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.893223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.928491 (+ 35268us) server_negotiation.cc:407] Connection header received
0926 07:34:03.928754 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.928761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.928851 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.928923 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.930867 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.932005 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.933755 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.934431 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.934537 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.938316 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.938325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.938331 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.938376 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.942056 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.942941 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.942945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.942948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.943108 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.943654 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.943661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.943666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.944074 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.944313 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.944893 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.945114 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:34:04.059154 29549 catalog_manager.cc:4841] TS 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773): tablet 50355f585e77484da8f9d0926d34a721 (table test-table [id=b0902d2ef50e4fb8b87f90e30deb0a35]) successfully deleted
I20260926 07:34:04.059484 29550 catalog_manager.cc:4841] TS 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679): tablet 50355f585e77484da8f9d0926d34a721 (table test-table [id=b0902d2ef50e4fb8b87f90e30deb0a35]) successfully deleted
W20260926 07:34:04.059715 29550 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.221.66:35679, user_credentials={real_user=slave}} blocked reactor thread for 140789us
I20260926 07:34:04.060070 29550 catalog_manager.cc:4841] TS 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833): tablet 50355f585e77484da8f9d0926d34a721 (table test-table [id=b0902d2ef50e4fb8b87f90e30deb0a35]) successfully deleted
W20260926 07:34:04.060354 29550 catalog_manager.cc:4808] TS 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833): delete failed for tablet 50355f585e77484da8f9d0926d34a721 because the tablet was not found. No further retry: Not found: Tablet not found: 50355f585e77484da8f9d0926d34a721
I20260926 07:34:04.060654 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 29531
I20260926 07:34:04.104439 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.086707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.22.221.125:44963)
0926 07:34:04.086929 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.086939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.088060 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 07:34:04.088336 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.088342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.088431 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.088510 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:04.091362 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.092524 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.094273 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.094850 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.094984 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.097884 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:04.097889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:04.097900 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:04.097976 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:04.101529 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:04.102201 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:04.102208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:04.102211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:04.102343 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:04.102812 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:04.102817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:04.102821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:04.103146 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:04.103393 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.103744 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.104085 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:04.105010 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.088451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.22.221.124:44905)
0926 07:34:04.088692 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.088700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.088716 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:04.088769 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.088774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.088852 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.088948 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:04.091489 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.092297 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.094230 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.094973 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.095099 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.098661 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:04.098669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:04.098679 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:04.098729 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:04.102002 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:04.102775 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:04.102782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:04.102785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:04.102903 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:04.103339 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:04.103345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:04.103348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:04.103722 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:04.103914 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.104493 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.104697 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:34:04.871515 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.870747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:35685)
0926 07:34:04.871178 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.871348 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
W20260926 07:34:04.871970 30673 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:42459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:04.881471 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.880477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:33449)
0926 07:34:04.881146 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.881287 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":401,"threads_started":1}
W20260926 07:34:04.881827 30814 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:42459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:04.894306 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.893652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:37361)
0926 07:34:04.894016 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.894158 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":181,"threads_started":1}
W20260926 07:34:04.894670 30527 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:42459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:05.389504 30882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02fa98fbcd424706b89819a21d1e0290)
I20260926 07:34:05.389791 30882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } }
I20260926 07:34:05.391137 30882 leader_election.cc:288] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459), 940f3dedb10544cea8abc0ba623aef3e (127.22.221.125:44963)
I20260926 07:34:05.393330 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.391463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40382)
0926 07:34:05.392942 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.393097 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1356,"mutex_wait_us":24,"thread_start_us":1160,"threads_started":1}
W20260926 07:34:05.393644 30332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:05.395282 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.394875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40392)
0926 07:34:05.395024 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.395120 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:34:05.395437 30332 leader_election.cc:334] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459): Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:05.396193 30887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02fa98fbcd424706b89819a21d1e0290)
I20260926 07:34:05.396402 30887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } }
I20260926 07:34:05.397656 30887 leader_election.cc:288] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459), 34d9ed909226465db4db83e3a38d6f3b (127.22.221.124:44905)
I20260926 07:34:05.401280 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.391460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.22.221.125:44963)
0926 07:34:05.393640 (+ 2180us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.393644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.393660 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:05.393717 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.393720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.393773 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.393894 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:05.394623 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.395791 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.397677 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.399567 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.399731 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.399841 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.400571 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.400774 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2087,"thread_start_us":1914,"threads_started":1}
I20260926 07:34:05.402424 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.391463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:44963 (local address 127.0.0.1:35094)
0926 07:34:05.392667 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.392697 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:05.392896 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:05.393935 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:05.393941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:05.393976 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:05.394431 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.394444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.395942 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.395951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.397491 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.397506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.400036 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.400044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.400311 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.400363 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:34:05.401755 (+ 1392us) client_negotiation.cc:240] Negotiation successful
0926 07:34:05.401959 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":836,"threads_started":1}
I20260926 07:34:05.403405 30171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34d9ed909226465db4db83e3a38d6f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "940f3dedb10544cea8abc0ba623aef3e" is_pre_election: true
I20260926 07:34:05.403807 30171 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d9ed909226465db4db83e3a38d6f3b in term 1.
I20260926 07:34:05.404498 30334 leader_election.cc:302] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34d9ed909226465db4db83e3a38d6f3b, 940f3dedb10544cea8abc0ba623aef3e; no voters: 02fa98fbcd424706b89819a21d1e0290
I20260926 07:34:05.405006 30882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:34:05.405195 30882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02fa98fbcd424706b89819a21d1e0290)
I20260926 07:34:05.405321 30882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:34:05.409415 30882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } }
I20260926 07:34:05.409986 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.397769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40406)
0926 07:34:05.409520 (+ 11751us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.409716 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11608,"threads_started":1}
I20260926 07:34:05.410729 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.410221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40422)
0926 07:34:05.410476 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.410596 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 07:34:05.410646 30171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34d9ed909226465db4db83e3a38d6f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "940f3dedb10544cea8abc0ba623aef3e"
I20260926 07:34:05.410868 30171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:34:05.410969 30882 leader_election.cc:288] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [CANDIDATE]: Term 2 election: Requested vote from peers 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459), 940f3dedb10544cea8abc0ba623aef3e (127.22.221.125:44963)
W20260926 07:34:05.411020 30145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:05.411890 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.411396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40436)
0926 07:34:05.411638 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.411730 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260926 07:34:05.412027 30332 leader_election.cc:334] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459): Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:05.413408 30171 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d9ed909226465db4db83e3a38d6f3b in term 2.
I20260926 07:34:05.413812 30334 leader_election.cc:302] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34d9ed909226465db4db83e3a38d6f3b, 940f3dedb10544cea8abc0ba623aef3e; no voters: 02fa98fbcd424706b89819a21d1e0290
I20260926 07:34:05.414287 30882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:34:05.415179 30882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [term 2 LEADER]: Becoming Leader. State: Replica: 34d9ed909226465db4db83e3a38d6f3b, State: Running, Role: LEADER
I20260926 07:34:05.415652 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.414951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40444)
0926 07:34:05.415183 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.415442 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:34:05.415727 30882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } }
W20260926 07:34:05.415846 30145 leader_election.cc:334] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459): Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
I20260926 07:34:05.417841 30892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d9ed909226465db4db83e3a38d6f3b. Latest consensus state: current_term: 2 leader_uuid: "34d9ed909226465db4db83e3a38d6f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:05.418277 30892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:05.418987 30894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:05.420439 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.397852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:44905 (local address 127.0.0.1:57900)
0926 07:34:05.410147 (+ 12295us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.410165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:05.410281 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:05.414882 (+ 4601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:05.414887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:05.414916 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:05.415273 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.415287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.416642 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.416648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.418130 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.418143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.419692 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.419699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.419878 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.419900 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:34:05.420007 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:34:05.420151 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12210,"thread_start_us":12494,"threads_started":1}
I20260926 07:34:05.420473 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.398268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.22.221.124:44905)
0926 07:34:05.414329 (+ 16061us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.414335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.414357 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:05.414432 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.414437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.414517 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.414629 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:05.415453 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.416499 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.418298 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.419504 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.419697 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.419772 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.420033 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.420160 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15905,"thread_start_us":15766,"threads_started":1}
I20260926 07:34:05.423048 30894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:05.423092 30358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "940f3dedb10544cea8abc0ba623aef3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34d9ed909226465db4db83e3a38d6f3b" is_pre_election: true
I20260926 07:34:05.423923 30147 leader_election.cc:302] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 940f3dedb10544cea8abc0ba623aef3e; no voters: 02fa98fbcd424706b89819a21d1e0290, 34d9ed909226465db4db83e3a38d6f3b
I20260926 07:34:05.424180 30894 catalog_manager.cc:1259] Loaded cluster ID: c259eddfa4434248a44bffc58578daae
I20260926 07:34:05.424258 30894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:05.424309 30887 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:34:05.425359 30894 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:05.426100 30894 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Loaded TSK: 0
I20260926 07:34:05.426834 30894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:05.434206 30347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:05.441058 30347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:05.444681 30171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [term 2 FOLLOWER]: Refusing update from remote peer 34d9ed909226465db4db83e3a38d6f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 07:34:05.444715 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.444383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40458)
0926 07:34:05.444532 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.444603 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:34:05.445456 30892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 07:34:05.445714 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.445315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40462)
0926 07:34:05.445488 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.445548 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:34:05.445888 30332 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b -> Peer 02fa98fbcd424706b89819a21d1e0290 (127.22.221.126:42459): Couldn't send request to peer 02fa98fbcd424706b89819a21d1e0290. Status: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:34:05.448094 30887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d9ed909226465db4db83e3a38d6f3b. Latest consensus state: current_term: 2 leader_uuid: "34d9ed909226465db4db83e3a38d6f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:05.448371 30887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:05.449575 30882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "34d9ed909226465db4db83e3a38d6f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:05.449857 30882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:05.450107 30896 mvcc.cc:205] Tried to move back new op lower bound from 7333511354135883776 to 7333511354026876928. Current Snapshot: MvccSnapshot[applied={T|T < 7333511354135883776}]
I20260926 07:34:05.450263 30887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "34d9ed909226465db4db83e3a38d6f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "02fa98fbcd424706b89819a21d1e0290" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42459 } } peers { permanent_uuid: "940f3dedb10544cea8abc0ba623aef3e" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 44963 } attrs { promote: false } } peers { permanent_uuid: "34d9ed909226465db4db83e3a38d6f3b" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 44905 } attrs { promote: false } } }
I20260926 07:34:05.450484 30887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940f3dedb10544cea8abc0ba623aef3e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:05.465728 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.458453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:32773 (local address 127.0.0.1:51272)
0926 07:34:05.458611 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.458627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:05.458747 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:05.459564 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:05.459568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:05.459594 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:05.459981 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.459992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.461427 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.461432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.463412 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.463426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.465006 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.465012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.465174 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.465209 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:34:05.465278 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:34:05.465407 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":20,"spinlock_wait_cycles":2688}
I20260926 07:34:05.467370 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.458813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.22.221.67:32773)
0926 07:34:05.459260 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.459267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.459288 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:34:05.459357 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.459362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.459435 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.460127 (+ 692us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:05.460301 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.461292 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.463621 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.464808 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.466742 (+ 1934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.466854 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.466908 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.467058 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":521,"threads_started":1}
I20260926 07:34:05.467969 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.458078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:35679 (local address 127.0.0.1:57560)
0926 07:34:05.458219 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.458239 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:05.458339 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:05.459515 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:05.459521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:05.459561 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:05.459780 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.459798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.461311 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.461316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.464759 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.464776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.467200 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.467205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.467432 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.467479 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:34:05.467580 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:34:05.467714 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1,"spinlock_wait_cycles":1792}
I20260926 07:34:05.468215 30748 tablet_service.cc:1467] Processing CreateTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be (DEFAULT_TABLE table=test-table [id=4feca140a203415e85852b30781ec097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:05.468560 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.458187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.22.221.66:35679)
0926 07:34:05.459045 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.459052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.459071 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:05.459145 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.459152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.459244 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.459347 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:05.459982 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.461156 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.464956 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.466879 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.467002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.467110 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.468146 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.468289 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":627,"threads_started":1}
I20260926 07:34:05.468670 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45ce7f4ebf44e2e83f511f06880d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:05.472132 30607 tablet_service.cc:1467] Processing CreateTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be (DEFAULT_TABLE table=test-table [id=4feca140a203415e85852b30781ec097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:05.472582 30607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45ce7f4ebf44e2e83f511f06880d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:05.473942 30904 tablet_bootstrap.cc:492] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Bootstrap starting.
I20260926 07:34:05.475899 30904 tablet_bootstrap.cc:654] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:05.477837 30904 tablet_bootstrap.cc:492] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: No bootstrap required, opened a new log
I20260926 07:34:05.477991 30904 ts_tablet_manager.cc:1397] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 07:34:05.478277 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.459155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:40833 (local address 127.0.0.1:43890)
0926 07:34:05.465942 (+ 6787us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.465959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:05.466046 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:05.472196 (+ 6150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:05.472201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:05.472229 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:05.472414 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.472423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.473798 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.473803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.475654 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:05.475668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.477521 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.477526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:05.477735 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.477793 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 07:34:05.477858 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:34:05.477984 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6685,"thread_start_us":12041,"threads_started":1}
I20260926 07:34:05.478739 30904 raft_consensus.cc:348] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.479089 30904 raft_consensus.cc:374] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:05.479271 30904 raft_consensus.cc:729] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38384ae8c9ad4ebc8b0e04c128a56039, State: Initialized, Role: FOLLOWER
I20260926 07:34:05.479336 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.459304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.22.221.65:40833)
0926 07:34:05.471867 (+ 12563us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.471873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.471892 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:05.471960 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.471964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.472053 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.472166 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:05.472568 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.473638 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.476096 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.477342 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.478582 (+ 1240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.478682 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.478736 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.478896 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12417,"thread_start_us":12327,"threads_started":1}
I20260926 07:34:05.479571 30904 consensus_queue.cc:260] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.480356 30906 tablet_bootstrap.cc:492] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Bootstrap starting.
I20260926 07:34:05.480305 30461 tablet_service.cc:1467] Processing CreateTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be (DEFAULT_TABLE table=test-table [id=4feca140a203415e85852b30781ec097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:05.480705 30461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45ce7f4ebf44e2e83f511f06880d5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:05.482834 30906 tablet_bootstrap.cc:654] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:05.485914 30909 tablet_bootstrap.cc:492] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Bootstrap starting.
I20260926 07:34:05.486716 30906 tablet_bootstrap.cc:492] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: No bootstrap required, opened a new log
I20260926 07:34:05.486903 30906 ts_tablet_manager.cc:1397] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 07:34:05.486958 30904 ts_tablet_manager.cc:1428] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Time spent starting tablet: real 0.009s user 0.001s sys 0.008s
I20260926 07:34:05.487460 30909 tablet_bootstrap.cc:654] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:05.488165 30906 raft_consensus.cc:348] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.488559 30906 raft_consensus.cc:374] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:05.488722 30906 raft_consensus.cc:729] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9aa2ed60d443c99b34d681bad4294, State: Initialized, Role: FOLLOWER
I20260926 07:34:05.489095 30906 consensus_queue.cc:260] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.489966 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.487488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:39935)
0926 07:34:05.489462 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.489648 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1856,"thread_start_us":1733,"threads_started":1}
I20260926 07:34:05.490223 30813 heartbeater.cc:503] Master 127.22.221.124:44905 was elected leader, sending a full tablet report...
I20260926 07:34:05.494249 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.490522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:50441)
0926 07:34:05.493689 (+ 3167us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.493920 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3005,"thread_start_us":2862,"threads_started":1}
I20260926 07:34:05.496968 30909 tablet_bootstrap.cc:492] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: No bootstrap required, opened a new log
I20260926 07:34:05.497188 30909 ts_tablet_manager.cc:1397] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260926 07:34:05.497400 30906 ts_tablet_manager.cc:1428] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260926 07:34:05.498157 30909 raft_consensus.cc:348] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.498420 30909 raft_consensus.cc:374] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:05.498548 30909 raft_consensus.cc:729] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64229b9e03ab45018198fe4555158212, State: Initialized, Role: FOLLOWER
I20260926 07:34:05.498987 30909 consensus_queue.cc:260] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.499306 30672 heartbeater.cc:503] Master 127.22.221.124:44905 was elected leader, sending a full tablet report...
I20260926 07:34:05.500444 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.497516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:48669)
0926 07:34:05.499819 (+ 2303us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.500089 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2124,"thread_start_us":2378,"threads_started":1}
I20260926 07:34:05.500864 30909 ts_tablet_manager.cc:1428] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 07:34:05.503187 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.502245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:41399)
0926 07:34:05.502763 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.502919 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":306,"threads_started":1}
I20260926 07:34:05.503510 30526 heartbeater.cc:503] Master 127.22.221.124:44905 was elected leader, sending a full tablet report...
I20260926 07:34:05.503595 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.501773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:48799)
0926 07:34:05.503090 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.503318 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1189,"threads_started":1}
I20260926 07:34:05.505283 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.504278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:34447)
0926 07:34:05.505058 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.505160 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677}
I20260926 07:34:05.597385 30907 raft_consensus.cc:482] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:05.597640 30907 raft_consensus.cc:504] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.598431 30907 leader_election.cc:288] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679), 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:05.598939 30627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45ce7f4ebf44e2e83f511f06880d5be" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9aa2ed60d443c99b34d681bad4294" is_pre_election: true
I20260926 07:34:05.598999 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45ce7f4ebf44e2e83f511f06880d5be" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64229b9e03ab45018198fe4555158212" is_pre_election: true
I20260926 07:34:05.599188 30627 raft_consensus.cc:2454] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 0.
I20260926 07:34:05.599212 30481 raft_consensus.cc:2454] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 0.
I20260926 07:34:05.599733 30704 leader_election.cc:302] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9aa2ed60d443c99b34d681bad4294, 38384ae8c9ad4ebc8b0e04c128a56039; no voters:
I20260926 07:34:05.599994 30907 raft_consensus.cc:2781] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:05.600109 30907 raft_consensus.cc:482] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:05.600245 30907 raft_consensus.cc:3037] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:05.602276 30907 raft_consensus.cc:504] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.602764 30907 leader_election.cc:288] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 election: Requested vote from peers 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679), 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833)
I20260926 07:34:05.603302 30627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45ce7f4ebf44e2e83f511f06880d5be" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9aa2ed60d443c99b34d681bad4294"
I20260926 07:34:05.603433 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45ce7f4ebf44e2e83f511f06880d5be" candidate_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64229b9e03ab45018198fe4555158212"
I20260926 07:34:05.603518 30627 raft_consensus.cc:3037] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:05.603670 30481 raft_consensus.cc:3037] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:05.605507 30627 raft_consensus.cc:2454] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 1.
I20260926 07:34:05.605561 30481 raft_consensus.cc:2454] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38384ae8c9ad4ebc8b0e04c128a56039 in term 1.
I20260926 07:34:05.605906 30704 leader_election.cc:302] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9aa2ed60d443c99b34d681bad4294, 38384ae8c9ad4ebc8b0e04c128a56039; no voters:
I20260926 07:34:05.606176 30907 raft_consensus.cc:2781] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:05.606632 30907 raft_consensus.cc:686] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 LEADER]: Becoming Leader. State: Replica: 38384ae8c9ad4ebc8b0e04c128a56039, State: Running, Role: LEADER
I20260926 07:34:05.606932 30907 consensus_queue.cc:237] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } }
I20260926 07:34:05.607676 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.607198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:57793)
0926 07:34:05.607366 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.607479 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:34:05.609783 30347 catalog_manager.cc:5495] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38384ae8c9ad4ebc8b0e04c128a56039" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 32773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:05.625578 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.620980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.22.221.67:32773)
0926 07:34:05.621165 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.621171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.621190 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:05.621410 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.621414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.621474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.621573 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:05.621934 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.622614 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.623974 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.624396 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.624527 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.624919 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:05.625009 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:05.625111 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.625259 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.625332 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:05.628491 30627 raft_consensus.cc:1261] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Refusing update from remote peer 38384ae8c9ad4ebc8b0e04c128a56039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:05.628491 30481 raft_consensus.cc:1261] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Refusing update from remote peer 38384ae8c9ad4ebc8b0e04c128a56039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:05.629078 30907 consensus_queue.cc:1035] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64229b9e03ab45018198fe4555158212" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:05.629350 30918 consensus_queue.cc:1035] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9aa2ed60d443c99b34d681bad4294" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:05.632817 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.632267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:43775)
0926 07:34:05.632480 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.632583 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 07:34:05.632848 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.631568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:35167)
0926 07:34:05.631766 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.631890 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 07:34:05.638106 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.637614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:42879)
0926 07:34:05.637858 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.637953 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 07:34:05.639389 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.638733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.66:58995)
0926 07:34:05.639011 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.639158 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 07:34:05.643003 30348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:05.643407 30348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:05.644723 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.644141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.67:52675)
0926 07:34:05.644391 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.644493 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 07:34:05.648047 30923 mvcc.cc:205] Tried to move back new op lower bound from 7333511354888179712 to 7333511354808688640. Current Snapshot: MvccSnapshot[applied={T|T < 7333511354888179712}]
I20260926 07:34:05.649570 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.649040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.22.221.65:38777)
0926 07:34:05.649288 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.649403 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 07:34:05.653076 30348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34d9ed909226465db4db83e3a38d6f3b: Sending DeleteTablet for 3 replicas of tablet d45ce7f4ebf44e2e83f511f06880d5be
I20260926 07:34:05.653900 30607 tablet_service.cc:1514] Processing DeleteTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:05 UTC) from {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:57560
I20260926 07:34:05.654008 30461 tablet_service.cc:1514] Processing DeleteTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:05 UTC) from {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:43890
I20260926 07:34:05.654579 30931 tablet_replica.cc:330] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: stopping tablet replica
I20260926 07:34:05.654759 30931 raft_consensus.cc:2229] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:05.654867 30748 tablet_service.cc:1514] Processing DeleteTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:05 UTC) from {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:51272
I20260926 07:34:05.654906 30460 tablet_service.cc:1514] Processing DeleteTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:05 UTC) from {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:43890
I20260926 07:34:05.655033 30931 raft_consensus.cc:2258] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:05.655298 30747 tablet_service.cc:1514] Processing DeleteTablet for tablet d45ce7f4ebf44e2e83f511f06880d5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:05 UTC) from {username='oryx', principal='oryx/127.22.221.124@KRBTEST.COM'} at 127.0.0.1:51272
I20260926 07:34:05.655633 30932 tablet_replica.cc:330] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: stopping tablet replica
I20260926 07:34:05.655817 30932 raft_consensus.cc:2229] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:05.656100 30932 raft_consensus.cc:2258] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:05.656220 30931 ts_tablet_manager.cc:1905] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:05.656255 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 30323
I20260926 07:34:05.656622 30933 tablet_replica.cc:330] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: stopping tablet replica
I20260926 07:34:05.656791 30933 raft_consensus.cc:2229] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:05.658281 30932 ts_tablet_manager.cc:1905] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:05.659272 30933 raft_consensus.cc:2258] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:05.660327 30933 ts_tablet_manager.cc:1905] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:05.662988 30931 ts_tablet_manager.cc:1918] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:05.663112 30931 log.cc:1196] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/d45ce7f4ebf44e2e83f511f06880d5be
I20260926 07:34:05.663583 30931 ts_tablet_manager.cc:1939] T d45ce7f4ebf44e2e83f511f06880d5be P 09c9aa2ed60d443c99b34d681bad4294: Deleting consensus metadata
I20260926 07:34:05.664599 30335 catalog_manager.cc:4841] TS 09c9aa2ed60d443c99b34d681bad4294 (127.22.221.66:35679): tablet d45ce7f4ebf44e2e83f511f06880d5be (table test-table [id=4feca140a203415e85852b30781ec097]) successfully deleted
I20260926 07:34:05.665259 30932 ts_tablet_manager.cc:1918] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:05.665378 30932 log.cc:1196] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/d45ce7f4ebf44e2e83f511f06880d5be
I20260926 07:34:05.665848 30932 ts_tablet_manager.cc:1939] T d45ce7f4ebf44e2e83f511f06880d5be P 64229b9e03ab45018198fe4555158212: Deleting consensus metadata
I20260926 07:34:05.666018 30933 ts_tablet_manager.cc:1918] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:05.666126 30933 log.cc:1196] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/d45ce7f4ebf44e2e83f511f06880d5be
I20260926 07:34:05.666419 30933 ts_tablet_manager.cc:1939] T d45ce7f4ebf44e2e83f511f06880d5be P 38384ae8c9ad4ebc8b0e04c128a56039: Deleting consensus metadata
I20260926 07:34:05.666882 30335 catalog_manager.cc:4841] TS 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833): tablet d45ce7f4ebf44e2e83f511f06880d5be (table test-table [id=4feca140a203415e85852b30781ec097]) successfully deleted
W20260926 07:34:05.667321 30335 catalog_manager.cc:4808] TS 64229b9e03ab45018198fe4555158212 (127.22.221.65:40833): delete failed for tablet d45ce7f4ebf44e2e83f511f06880d5be because the tablet was not found. No further retry: Not found: Tablet not found: d45ce7f4ebf44e2e83f511f06880d5be
I20260926 07:34:05.667371 30334 catalog_manager.cc:4841] TS 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773): tablet d45ce7f4ebf44e2e83f511f06880d5be (table test-table [id=4feca140a203415e85852b30781ec097]) successfully deleted
W20260926 07:34:05.667894 30334 catalog_manager.cc:4808] TS 38384ae8c9ad4ebc8b0e04c128a56039 (127.22.221.67:32773): delete failed for tablet d45ce7f4ebf44e2e83f511f06880d5be because the tablet was not found. No further retry: Not found: Tablet not found: d45ce7f4ebf44e2e83f511f06880d5be
I20260926 07:34:05.674455 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.657281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.22.221.65:40833)
0926 07:34:05.657476 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.657482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.659175 (+ 1693us) server_negotiation.cc:407] Connection header received
0926 07:34:05.659380 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.659386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.659455 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.659528 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:05.661483 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.662593 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.664593 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.665332 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.665444 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.669384 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:05.669396 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:05.669410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:05.669440 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:05.672130 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:05.672767 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:05.672774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:05.672776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:05.672910 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:05.673261 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:05.673264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:05.673266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:05.673584 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:05.673796 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.673985 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.674226 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:34:05.675029 30521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:05.767611 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 30323
I20260926 07:34:05.784951 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 30538
I20260926 07:34:05.798167 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.785747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.22.221.66:35679)
0926 07:34:05.785946 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.785951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.785970 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:05.786111 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.786115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.786183 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.786264 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:05.787590 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.788285 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.789499 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.789963 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.790065 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.792642 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:05.792656 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:05.792672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:05.792702 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:05.795690 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:05.796332 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:05.796339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:05.796341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:05.796464 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:05.796787 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:05.796790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:05.796792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:05.797125 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:05.797325 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.797571 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.797820 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:34:05.798895 30667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:05.892138 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 30538
I20260926 07:34:05.904647 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.904197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40468)
0926 07:34:05.904399 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.904483 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:34:05.905679 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.905368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42459 (local address 127.0.0.1:40474)
0926 07:34:05.905511 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:05.905580 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:34:05.910494 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 30685
I20260926 07:34:05.922811 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:05.911169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.22.221.67:32773)
0926 07:34:05.911348 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:05.911352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:05.911369 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:34:05.911498 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:05.911502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:05.911603 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:05.911687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:05.912935 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.913605 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.914722 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:05.915081 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:05.915172 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:05.917802 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:05.917815 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:05.917830 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:05.917865 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:05.920532 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:05.921201 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:05.921208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:05.921210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:05.921337 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:05.921659 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:05.921662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:05.921664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:05.921988 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:05.922205 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:05.922410 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:34:05.922568 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:34:05.923703 30808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:06.044196 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 30685
I20260926 07:34:06.065580 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 30024
I20260926 07:34:06.066694 30191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:06.182621 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 30024
I20260926 07:34:06.203469 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 30124
I20260926 07:34:06.204514 30378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:06.324402 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 30124
2026-09-26T07:34:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9153 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:06 dist-test-slave-qlq8 krb5kdc[30942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:06 dist-test-slave-qlq8 krb5kdc[30942](info): set up 2 sockets
Sep 26 07:34:06 dist-test-slave-qlq8 krb5kdc[30942](info): commencing operation
krb5kdc: starting...
W20260926 07:34:07.440248 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:34:07 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab.
Entry for principal oryx/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab.
I20260926 07:34:07.647682 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35765
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab
--principal=oryx/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:07.963627 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:07.963884 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:07.964061 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:07.971349 30955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:07.971464 30955 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:34:07.971547 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:07.971660 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:07.971753 30955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:07.971843 30955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:07.981441 30955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35765
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.126.keytab
--principal=oryx/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:07.982676 30955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:07.984022 30955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:07.995936 30960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.995893 30961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.995965 30963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.996726 30955 server_base.cc:1034] running on GCE node
I20260926 07:34:07.997438 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:07.998739 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:07.999950 30955 hybrid_clock.cc:648] HybridClock initialized: now 1790408047999894 us; error 43 us; skew 500 ppm
I20260926 07:34:08.000286 30955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:07 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408047, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:08.003315 30955 init.cc:377] Logged in from keytab as oryx/127.22.221.126@KRBTEST.COM (short username oryx)
I20260926 07:34:08.004740 30955 webserver.cc:460] Webserver started at http://127.22.221.126:40653/ using document root <none> and password file <none>
I20260926 07:34:08.005422 30955 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:08.005560 30955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:08.005884 30955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:08.009328 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "5885682c4e6140b0be8bbf3fd1dbcd26"
format_stamp: "Formatted at 2026-09-26 07:34:08 on dist-test-slave-qlq8"
server_key: "2f09417327524b7727766546f69c88aa"
server_key_iv: "7527574d9cb024b4b8955d8b2fd46e96"
server_key_version: "encryptionkey@0"
I20260926 07:34:08.010118 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "5885682c4e6140b0be8bbf3fd1dbcd26"
format_stamp: "Formatted at 2026-09-26 07:34:08 on dist-test-slave-qlq8"
server_key: "2f09417327524b7727766546f69c88aa"
server_key_iv: "7527574d9cb024b4b8955d8b2fd46e96"
server_key_version: "encryptionkey@0"
I20260926 07:34:08.014998 30955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 07:34:08.018208 30970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:08.019502 30955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:08.019687 30955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "5885682c4e6140b0be8bbf3fd1dbcd26"
format_stamp: "Formatted at 2026-09-26 07:34:08 on dist-test-slave-qlq8"
server_key: "2f09417327524b7727766546f69c88aa"
server_key_iv: "7527574d9cb024b4b8955d8b2fd46e96"
server_key_version: "encryptionkey@0"
I20260926 07:34:08.019897 30955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:08.035401 30955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:08.039124 30955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:08.039422 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:08.060853 30955 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:35765
I20260926 07:34:08.060859 31031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:35765 every 8 connection(s)
I20260926 07:34:08.062171 30955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:08.065806 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:08.067052 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 30955
I20260926 07:34:08.067291 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:34:08.068105 23413 external_mini_cluster.cc:1428] Setting key 05236b590d78615d0d5c4f6cdcb6a280
I20260926 07:34:08.080205 31032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Bootstrap starting.
Sep 26 07:34:08 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:08.084131 31032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:08.085301 31032 log.cc:824] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:08.088061 31032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: No bootstrap required, opened a new log
I20260926 07:34:08.090785 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.070208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.22.221.126:35765)
0926 07:34:08.071004 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.071014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.071056 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:34:08.072678 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.072699 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.073108 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.073417 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:08.075063 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.076379 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.078161 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.078770 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.078888 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.083874 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:08.083881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:08.083894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:08.083925 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:08.086926 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:08.087654 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:08.087667 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:08.087673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:08.087837 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:08.088231 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:08.088236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:08.088239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:08.088704 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:08.089022 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.089434 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.089834 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":365,"threads_started":1}
I20260926 07:34:08.096202 31032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35765 } }
I20260926 07:34:08.096544 31032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:08.096647 31032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5885682c4e6140b0be8bbf3fd1dbcd26, State: Initialized, Role: FOLLOWER
I20260926 07:34:08.097242 31032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35765 } }
I20260926 07:34:08.097529 31032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:08.097669 31032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:08.097855 31032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:08.099944 31032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35765 } }
I20260926 07:34:08.100414 31032 leader_election.cc:302] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5885682c4e6140b0be8bbf3fd1dbcd26; no voters:
I20260926 07:34:08.100953 31032 leader_election.cc:288] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:08.101076 31037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:08.101780 31037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [term 1 LEADER]: Becoming Leader. State: Replica: 5885682c4e6140b0be8bbf3fd1dbcd26, State: Running, Role: LEADER
I20260926 07:34:08.102375 31037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35765 } }
I20260926 07:34:08.102988 31032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:08.105270 31039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5885682c4e6140b0be8bbf3fd1dbcd26. Latest consensus state: current_term: 1 leader_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35765 } } }
I20260926 07:34:08.105913 31039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:08.106016 31038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5885682c4e6140b0be8bbf3fd1dbcd26" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35765 } } }
I20260926 07:34:08.106308 31038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:08.112141 31045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:08.116406 31045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:08.124439 31045 catalog_manager.cc:1347] Generated new cluster ID: 32c639c2647c45e38854445cb07ec0bf
I20260926 07:34:08.124557 31045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:08.139137 31045 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:08.140964 31045 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:08.150089 31045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Generated new TSK 0
I20260926 07:34:08.151046 31045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab.
Entry for principal oryx/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab.
I20260926 07:34:08.201143 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35765
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:08.525130 31058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:08.525395 31058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:08.525491 31058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:08.532526 31058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:08.532622 31058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:08.532702 31058 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:34:08.532802 31058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:08.542058 31058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:08.543380 31058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:08.544624 31058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:08.557226 31063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:08.557211 31064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:08.557312 31066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:08.558108 31058 server_base.cc:1034] running on GCE node
I20260926 07:34:08.558683 31058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:08.559620 31058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:08.560870 31058 hybrid_clock.cc:648] HybridClock initialized: now 1790408048560769 us; error 65 us; skew 500 ppm
I20260926 07:34:08.561154 31058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:08 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408048, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:08.564349 31058 init.cc:377] Logged in from keytab as oryx/127.22.221.65@KRBTEST.COM (short username oryx)
I20260926 07:34:08.565743 31058 webserver.cc:460] Webserver started at http://127.22.221.65:35563/ using document root <none> and password file <none>
I20260926 07:34:08.566433 31058 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:08.566560 31058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:08.566879 31058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:08.569672 31058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "ac8b37728c944a73bafb9edc1d10d766"
format_stamp: "Formatted at 2026-09-26 07:34:08 on dist-test-slave-qlq8"
server_key: "acd884c86fafd57f30690af30b34e65b"
server_key_iv: "ff8be022a2fac3d8e0341564e16e0238"
server_key_version: "encryptionkey@0"
I20260926 07:34:08.570607 31058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "ac8b37728c944a73bafb9edc1d10d766"
format_stamp: "Formatted at 2026-09-26 07:34:08 on dist-test-slave-qlq8"
server_key: "acd884c86fafd57f30690af30b34e65b"
server_key_iv: "ff8be022a2fac3d8e0341564e16e0238"
server_key_version: "encryptionkey@0"
I20260926 07:34:08.575508 31058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:34:08.578687 31073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:08.580021 31058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:08.580192 31058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "ac8b37728c944a73bafb9edc1d10d766"
format_stamp: "Formatted at 2026-09-26 07:34:08 on dist-test-slave-qlq8"
server_key: "acd884c86fafd57f30690af30b34e65b"
server_key_iv: "ff8be022a2fac3d8e0341564e16e0238"
server_key_version: "encryptionkey@0"
I20260926 07:34:08.580400 31058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:08.596590 31058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:08.600698 31058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:08.600955 31058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:08.603621 31058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:08.603720 31058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:08.603869 31058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:08.603979 31058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 07:34:08 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408048, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:08.625751 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.604112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:35157 (local address 127.22.221.126:35765)
0926 07:34:08.604855 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.604864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.605701 (+ 837us) server_negotiation.cc:407] Connection header received
0926 07:34:08.608008 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.608015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.608100 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.608225 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:08.610189 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.611275 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.613835 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.614455 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.614597 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.619225 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:08.619232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:08.619238 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:08.619305 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:08.622462 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:08.623346 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:08.623353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:08.623357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:08.623494 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:08.623991 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:08.623995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:08.623998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:08.624330 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:08.624485 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.625150 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.625365 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":508,"threads_started":1}
I20260926 07:34:08.627028 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.604768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.22.221.65:35157)
0926 07:34:08.605556 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.605591 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.607656 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.608455 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.608463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.608957 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:08.609924 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.609955 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.611408 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.611413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.613684 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.613707 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.614720 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.614727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.614961 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.615711 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:08.615731 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:08.618977 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:08.622602 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:08.622610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:08.622625 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:08.623196 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:08.623618 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:08.623623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:08.623626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:08.623821 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:08.624458 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:08.624466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:08.624795 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.625053 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.625540 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":311,"threads_started":1}
I20260926 07:34:08.649703 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.633571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:50795 (local address 127.22.221.126:35765)
0926 07:34:08.633882 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.633886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.634451 (+ 565us) server_negotiation.cc:407] Connection header received
0926 07:34:08.634642 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.634646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.634703 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.634781 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:08.636486 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.637726 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.640537 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.641086 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.641186 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.643876 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:08.643881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:08.643885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:08.643914 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:08.646407 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:08.647395 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:08.647403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:08.647409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:08.647602 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:08.647985 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:08.647992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:08.647997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:08.648392 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:08.648559 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.649085 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.649314 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260926 07:34:08.649820 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.633605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.22.221.65:50795)
0926 07:34:08.634315 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.634340 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.634523 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.634846 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.634855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.635597 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:08.636276 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.636293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.637970 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.637977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.640285 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.640307 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.641292 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.641302 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.641549 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.642267 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:08.642298 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:08.643679 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:08.646548 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:08.646556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:08.646563 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:08.647254 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:08.647708 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:08.647713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:08.647717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:08.647859 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:08.648521 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:08.648526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:08.648771 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.648972 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.649328 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":349,"threads_started":1}
I20260926 07:34:08.658201 31058 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:44073
I20260926 07:34:08.658243 31193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:44073 every 8 connection(s)
I20260926 07:34:08.659412 31058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:08.661237 31194 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35765
I20260926 07:34:08.661455 31194 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:08.661707 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 31058
I20260926 07:34:08.661917 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:08.662204 31194 heartbeater.cc:511] Master 127.22.221.126:35765 requested a full tablet report, sending...
I20260926 07:34:08.662667 23413 external_mini_cluster.cc:1428] Setting key 86f2aee24585ff551a4320d9211ecc71
I20260926 07:34:08.664253 30987 ts_manager.cc:194] Registered new tserver with Master: ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073)
I20260926 07:34:08.665854 30987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:35157
WARNING: no policy specified for oryx/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab.
Entry for principal oryx/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab.
I20260926 07:34:08.708709 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35765
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:09.027751 31198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:09.028018 31198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:09.028115 31198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:09.035077 31198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:09.035178 31198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:09.035284 31198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:34:09.035367 31198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:09.044875 31198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:09.046063 31198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:09.047477 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:09.060539 31204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:09.060811 31203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:09.061146 31206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:09.061546 31198 server_base.cc:1034] running on GCE node
I20260926 07:34:09.062233 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:09.063361 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:09.064608 31198 hybrid_clock.cc:648] HybridClock initialized: now 1790408049064527 us; error 54 us; skew 500 ppm
I20260926 07:34:09.064944 31198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:09.068763 31198 init.cc:377] Logged in from keytab as oryx/127.22.221.66@KRBTEST.COM (short username oryx)
I20260926 07:34:09.072439 31198 webserver.cc:460] Webserver started at http://127.22.221.66:43833/ using document root <none> and password file <none>
I20260926 07:34:09.073206 31198 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:09.073357 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:09.073696 31198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:09.076781 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "c1eac5119d1c4e6b9c3e46331708d5dc"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-qlq8"
server_key: "68c35aaefc26c77e69bc6d6729063acf"
server_key_iv: "15dd36db254fca719922adc98cd8e1e5"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.077682 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "c1eac5119d1c4e6b9c3e46331708d5dc"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-qlq8"
server_key: "68c35aaefc26c77e69bc6d6729063acf"
server_key_iv: "15dd36db254fca719922adc98cd8e1e5"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.083303 31198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:34:09.086860 31213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:09.088174 31198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:09.088353 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "c1eac5119d1c4e6b9c3e46331708d5dc"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-qlq8"
server_key: "68c35aaefc26c77e69bc6d6729063acf"
server_key_iv: "15dd36db254fca719922adc98cd8e1e5"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.088583 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:09.104142 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:09.108552 31198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:09.108855 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:09.111639 31198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:09.111747 31198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:09.111898 31198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:09.111987 31198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:09.136358 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.112211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:42959 (local address 127.22.221.126:35765)
0926 07:34:09.112443 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.112450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.113919 (+ 1469us) server_negotiation.cc:407] Connection header received
0926 07:34:09.116055 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.116064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.116178 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.116308 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.118277 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.119796 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.122205 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.122892 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.123043 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.128224 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:09.128235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:09.128242 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:09.128298 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:09.132816 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.133572 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.133582 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.133588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.133790 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.134265 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.134275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.134281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.134827 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:09.135047 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.135775 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.136000 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:34:09.138051 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.112946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.22.221.66:42959)
0926 07:34:09.113756 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.113794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.115657 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.116542 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.116557 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.117323 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:09.118055 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.118089 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.119991 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.119995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.122053 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.122064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.123156 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.123165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.123490 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.124303 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:09.124343 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:09.127942 (+ 3599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:09.132999 (+ 5057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.133007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.133020 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.133441 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.133956 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.133959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.133961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.134153 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.134987 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:09.134993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:09.135387 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.135681 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.136250 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":368,"threads_started":1}
I20260926 07:34:09.161338 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.145539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.22.221.66:35865)
0926 07:34:09.146444 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.146475 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.146662 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.147034 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.147042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.147877 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:09.148400 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.148417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.149425 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.149433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.151862 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.151883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.152926 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.152936 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.153088 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.153946 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:09.153975 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:09.154762 (+ 787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:09.158446 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.158451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.158455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.158843 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.159219 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.159225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.159258 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.159390 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.160114 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:09.160119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:09.160370 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.160553 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.160837 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":488,"threads_started":1}
I20260926 07:34:09.161589 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.145421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:35865 (local address 127.22.221.126:35765)
0926 07:34:09.145616 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.145622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.146585 (+ 963us) server_negotiation.cc:407] Connection header received
0926 07:34:09.146803 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.146808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.146879 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.146959 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.148561 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.149273 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.152046 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.152685 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.152831 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.155014 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:09.155020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:09.155024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:09.155062 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:09.158249 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.158960 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.158967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.158971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.159107 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.159502 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.159507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.159514 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.159981 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:09.160150 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.161032 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.161243 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:09.165545 31198 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:39661
I20260926 07:34:09.165565 31332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:39661 every 8 connection(s)
I20260926 07:34:09.166882 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:09.168895 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 31198
I20260926 07:34:09.169070 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:09.169561 23413 external_mini_cluster.cc:1428] Setting key 42e97084d60ced544396474d032c10e5
I20260926 07:34:09.170060 31333 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35765
I20260926 07:34:09.170320 31333 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:09.171063 31333 heartbeater.cc:511] Master 127.22.221.126:35765 requested a full tablet report, sending...
I20260926 07:34:09.176680 30987 ts_manager.cc:194] Registered new tserver with Master: c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661)
I20260926 07:34:09.177690 30987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:42959
WARNING: no policy specified for oryx/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab.
Entry for principal oryx/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab.
I20260926 07:34:09.216421 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35765
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:09.533564 31337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:09.533811 31337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:09.533915 31337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:09.540735 31337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:09.540831 31337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:09.540916 31337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:34:09.540992 31337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:09.550067 31337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.221.126:35765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:09.551126 31337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:09.552322 31337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:09.564524 31342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:09.564576 31343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:09.564600 31345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:09.565181 31337 server_base.cc:1034] running on GCE node
I20260926 07:34:09.565727 31337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:09.566599 31337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:09.567804 31337 hybrid_clock.cc:648] HybridClock initialized: now 1790408049567751 us; error 36 us; skew 500 ppm
I20260926 07:34:09.568089 31337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:09.570991 31337 init.cc:377] Logged in from keytab as oryx/127.22.221.67@KRBTEST.COM (short username oryx)
I20260926 07:34:09.572404 31337 webserver.cc:460] Webserver started at http://127.22.221.67:33621/ using document root <none> and password file <none>
I20260926 07:34:09.573104 31337 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:09.573235 31337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:09.573550 31337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:09.576412 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "98111d5a00bf4364a9d14560e5ea6a58"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-qlq8"
server_key: "699fb4a6ea895c407316466ba082367b"
server_key_iv: "8dd5e465b04624eff6588a6d6f484f10"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.577173 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "98111d5a00bf4364a9d14560e5ea6a58"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-qlq8"
server_key: "699fb4a6ea895c407316466ba082367b"
server_key_iv: "8dd5e465b04624eff6588a6d6f484f10"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.581924 31337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 07:34:09.585280 31352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:09.586637 31337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:09.586793 31337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "98111d5a00bf4364a9d14560e5ea6a58"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-qlq8"
server_key: "699fb4a6ea895c407316466ba082367b"
server_key_iv: "8dd5e465b04624eff6588a6d6f484f10"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.586997 31337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:09.600600 31337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:09.604524 31337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:09.604790 31337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:09.607357 31337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:09.607434 31337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:09.607584 31337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:09.607724 31337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:09.630801 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.607866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:34197 (local address 127.22.221.126:35765)
0926 07:34:09.608050 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.608056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.609486 (+ 1430us) server_negotiation.cc:407] Connection header received
0926 07:34:09.611796 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.611802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.611887 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.611979 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.614359 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.615106 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.617760 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.618308 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.618441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.623148 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:09.623157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:09.623164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:09.623209 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:09.626564 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.627419 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.627424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.627428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.627572 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.628267 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.628272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.628276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.628624 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:09.628808 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.630067 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.630336 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:34:09.632725 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.608460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.22.221.67:34197)
0926 07:34:09.609313 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.609357 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.611443 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.612514 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.612525 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.613336 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:09.614196 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.614215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.615293 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.615302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.617601 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.617616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.618551 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.618559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.618794 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.619733 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:09.619774 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:09.622862 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:09.626731 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.626737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.626747 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.627296 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.627684 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.627689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.627693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.628080 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.628801 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:09.628816 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:09.629432 (+ 616us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.629908 (+ 476us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.630614 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":366,"threads_started":1}
I20260926 07:34:09.650048 31337 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:35345
I20260926 07:34:09.650141 31471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:35345 every 8 connection(s)
I20260926 07:34:09.651573 31337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:09.654007 31472 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35765
I20260926 07:34:09.654237 31472 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:09.654944 31472 heartbeater.cc:511] Master 127.22.221.126:35765 requested a full tablet report, sending...
I20260926 07:34:09.655409 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.640337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:40169 (local address 127.22.221.126:35765)
0926 07:34:09.640551 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.640556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.640699 (+ 143us) server_negotiation.cc:407] Connection header received
0926 07:34:09.640875 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.640884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.640997 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.641116 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.642141 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.643542 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.646432 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.646898 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.647053 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.649233 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:09.649239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:09.649245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:09.649284 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:09.652157 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.653134 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.653139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.653143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.653328 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.653720 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.653726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.653730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.654124 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:09.654272 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.654743 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.654985 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:34:09.655539 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.639876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.22.221.67:40169)
0926 07:34:09.640578 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.640598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.640733 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.641155 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.641158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.641631 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:09.641974 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.641982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.643742 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.643751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.646250 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.646267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.647342 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.647349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.647512 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.648212 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:09.648229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:09.648964 (+ 735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:09.652301 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.652306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.652310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.653024 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.653440 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:09.653444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:09.653447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:09.653599 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:09.654250 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:09.654254 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:09.654455 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.654640 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.655000 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":683,"threads_started":1}
I20260926 07:34:09.656611 30986 ts_manager.cc:194] Registered new tserver with Master: 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:09.657429 30986 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:34197
I20260926 07:34:09.657896 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 31337
I20260926 07:34:09.658078 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:09.658511 23413 external_mini_cluster.cc:1428] Setting key 43b59e8cc0a3766a593c6c418aa81c51
I20260926 07:34:09.661438 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:09.668473 31194 heartbeater.cc:503] Master 127.22.221.126:35765 was elected leader, sending a full tablet report...
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:34:09.709481 31082 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:09.699083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.22.221.126:35765)
0926 07:34:09.699343 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.699351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.699602 (+ 251us) server_negotiation.cc:407] Connection header received
0926 07:34:09.699822 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.699826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.699888 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.699996 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.701279 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.702031 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.703289 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.703734 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.703869 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.709157 (+ 5288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43678: BlockingRecv error: recv got EOF from 127.0.0.1:43678 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 07:34:09 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:09.724442 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.712483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.22.221.126:35765)
0926 07:34:09.712672 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.712679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.712848 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:34:09.712959 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.712962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.713030 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.713109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.714203 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.714881 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.716104 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.716479 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.716574 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.719867 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:09.719872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:09.719875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:09.719902 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:09.722162 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.722774 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.722778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.722781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.722893 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.723221 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.723224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.723268 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.723536 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:09.723831 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.723963 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.724122 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:34:09.727921 30986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:09.734781 30986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:09.761638 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.746563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:44073 (local address 127.0.0.1:36224)
0926 07:34:09.749698 (+ 3135us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.749719 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.749893 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.750851 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.750862 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.750904 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:09.751413 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.751430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.754044 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.754052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.757056 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.757071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.760787 (+ 3716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.760791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.760928 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.760960 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.761093 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.761287 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3015,"thread_start_us":1982,"threads_started":1}
I20260926 07:34:09.761638 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.746817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.22.221.65:44073)
0926 07:34:09.748350 (+ 1533us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.748362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.749802 (+ 1440us) server_negotiation.cc:407] Connection header received
0926 07:34:09.750084 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.750092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.750374 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.750553 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:09.751675 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.753389 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.757229 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.759146 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.759316 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.759601 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.761154 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.761324 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":1134,"threads_started":1}
I20260926 07:34:09.765820 31124 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3482d8b32746d4b1366e53d94fb94d (DEFAULT_TABLE table=test-table [id=083b1ac2726d47499c8feac17d90656b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:09.767323 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3482d8b32746d4b1366e53d94fb94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:09.770128 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.749920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:35345 (local address 127.0.0.1:55628)
0926 07:34:09.759757 (+ 9837us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.759782 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.759928 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.761921 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.761928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.761964 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:09.763902 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.763915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.765714 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.765721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.767334 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.767347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.769262 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.769269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.769449 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.769481 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.769570 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.769706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9674,"mutex_wait_us":9,"thread_start_us":10117,"threads_started":1}
I20260926 07:34:09.770152 31495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.750596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.22.221.67:35345)
0926 07:34:09.760897 (+ 10301us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.760909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.760954 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:34:09.761090 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.761102 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.761420 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.761660 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:09.764133 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.765587 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.767590 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.769039 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.769173 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.769413 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.769660 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.769827 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10074,"thread_start_us":9822,"threads_started":1}
I20260926 07:34:09.770977 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.746222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39661 (local address 127.0.0.1:39310)
0926 07:34:09.748897 (+ 2675us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.748937 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.749090 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.762974 (+ 13884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.762979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.763005 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:09.763573 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.763587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.765313 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.765318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.767473 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.767494 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.770301 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.770307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.770471 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.770493 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.770584 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.770724 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2446,"mutex_wait_us":19,"thread_start_us":3188,"threads_started":1}
I20260926 07:34:09.771885 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.746422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.22.221.66:39661)
0926 07:34:09.762519 (+ 16097us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.762526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.762548 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:09.762622 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.762628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.762793 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.762933 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:09.763729 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.765152 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.767647 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.770075 (+ 2428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.771215 (+ 1140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.771415 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.771475 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.771590 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15948,"thread_start_us":15837,"threads_started":1}
I20260926 07:34:09.774516 31402 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3482d8b32746d4b1366e53d94fb94d (DEFAULT_TABLE table=test-table [id=083b1ac2726d47499c8feac17d90656b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:09.775496 31263 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3482d8b32746d4b1366e53d94fb94d (DEFAULT_TABLE table=test-table [id=083b1ac2726d47499c8feac17d90656b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:09.776057 31402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3482d8b32746d4b1366e53d94fb94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:09.776793 31263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3482d8b32746d4b1366e53d94fb94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:09.779986 31496 tablet_bootstrap.cc:492] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Bootstrap starting.
I20260926 07:34:09.782974 31496 tablet_bootstrap.cc:654] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:09.784169 31496 log.cc:824] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:09.787360 31496 tablet_bootstrap.cc:492] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: No bootstrap required, opened a new log
I20260926 07:34:09.787629 31496 ts_tablet_manager.cc:1397] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:34:09.790978 31498 tablet_bootstrap.cc:492] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Bootstrap starting.
I20260926 07:34:09.791376 31499 tablet_bootstrap.cc:492] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Bootstrap starting.
I20260926 07:34:09.794759 31499 tablet_bootstrap.cc:654] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:09.796000 31499 log.cc:824] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:09.797587 31496 raft_consensus.cc:348] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.798135 31498 tablet_bootstrap.cc:654] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:09.798130 31496 raft_consensus.cc:374] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:09.798238 31496 raft_consensus.cc:729] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8b37728c944a73bafb9edc1d10d766, State: Initialized, Role: FOLLOWER
I20260926 07:34:09.799188 31498 log.cc:824] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:09.799012 31496 consensus_queue.cc:260] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.800868 31496 ts_tablet_manager.cc:1428] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 07:34:09.802752 31499 tablet_bootstrap.cc:492] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: No bootstrap required, opened a new log
I20260926 07:34:09.802999 31499 ts_tablet_manager.cc:1397] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 07:34:09.803759 31498 tablet_bootstrap.cc:492] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: No bootstrap required, opened a new log
I20260926 07:34:09.804040 31498 ts_tablet_manager.cc:1397] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260926 07:34:09.811151 31499 raft_consensus.cc:348] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.811568 31499 raft_consensus.cc:374] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:09.811679 31499 raft_consensus.cc:729] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Initialized, Role: FOLLOWER
I20260926 07:34:09.811964 31498 raft_consensus.cc:348] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.812279 31498 raft_consensus.cc:374] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:09.812369 31498 raft_consensus.cc:729] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Initialized, Role: FOLLOWER
I20260926 07:34:09.812259 31499 consensus_queue.cc:260] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.812912 31498 consensus_queue.cc:260] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.813333 31333 heartbeater.cc:503] Master 127.22.221.126:35765 was elected leader, sending a full tablet report...
I20260926 07:34:09.813939 31472 heartbeater.cc:503] Master 127.22.221.126:35765 was elected leader, sending a full tablet report...
I20260926 07:34:09.813977 31499 ts_tablet_manager.cc:1428] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 07:34:09.814539 31498 ts_tablet_manager.cc:1428] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
W20260926 07:34:09.903673 31473 tablet.cc:2369] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:34:09.911672 31195 tablet.cc:2369] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:09.916697 31504 raft_consensus.cc:482] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:09.917038 31504 raft_consensus.cc:504] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
W20260926 07:34:09.919140 31334 tablet.cc:2369] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:09.918716 31504 leader_election.cc:288] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661), ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073)
I20260926 07:34:09.924624 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.918986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39661 (local address 127.22.221.67:60983)
0926 07:34:09.919568 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.919593 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.919738 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.920143 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.920152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.920178 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:09.920383 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.920394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.921395 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.921400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.922705 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.922716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.923910 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.923916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.924103 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.924126 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.924231 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.924362 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260926 07:34:09.924631 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.919157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:60983 (local address 127.22.221.66:39661)
0926 07:34:09.919396 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.919401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.919674 (+ 273us) server_negotiation.cc:407] Connection header received
0926 07:34:09.919937 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.919943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.920025 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.920126 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:09.920535 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.921237 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.922860 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.923756 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.923923 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.923996 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.924255 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.924374 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:34:09.925573 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3482d8b32746d4b1366e53d94fb94d" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" is_pre_election: true
I20260926 07:34:09.926041 31283 raft_consensus.cc:2454] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 0.
I20260926 07:34:09.926088 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.919371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:56507 (local address 127.22.221.65:44073)
0926 07:34:09.919590 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.919598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.920702 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 07:34:09.920896 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.920899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.920966 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.921070 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:09.921662 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.922346 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.924132 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.925203 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.925374 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.925446 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.925728 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.925848 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:34:09.926126 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.919770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:44073 (local address 127.22.221.67:56507)
0926 07:34:09.920598 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:09.920611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:09.920748 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:09.921092 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:09.921096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:09.921120 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:09.921482 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.921494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.922484 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.922490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.923976 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:09.923988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.925410 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.925417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:09.925575 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.925598 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:34:09.925703 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:34:09.925820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":508,"threads_started":1}
I20260926 07:34:09.926730 31355 leader_election.cc:302] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:09.927137 31504 raft_consensus.cc:2781] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:09.927312 31504 raft_consensus.cc:482] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:09.927161 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3482d8b32746d4b1366e53d94fb94d" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766" is_pre_election: true
I20260926 07:34:09.927429 31504 raft_consensus.cc:3037] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:09.927685 31149 raft_consensus.cc:2454] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 0.
I20260926 07:34:09.929807 31504 raft_consensus.cc:504] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.930398 31504 leader_election.cc:288] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 election: Requested vote from peers c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661), ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073)
I20260926 07:34:09.930824 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3482d8b32746d4b1366e53d94fb94d" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc"
I20260926 07:34:09.931021 31283 raft_consensus.cc:3037] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:09.931416 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3482d8b32746d4b1366e53d94fb94d" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766"
I20260926 07:34:09.931610 31149 raft_consensus.cc:3037] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:09.933305 31283 raft_consensus.cc:2454] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 1.
I20260926 07:34:09.933670 31355 leader_election.cc:302] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:09.933921 31149 raft_consensus.cc:2454] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 1.
I20260926 07:34:09.933930 31504 raft_consensus.cc:2781] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:09.934708 31504 raft_consensus.cc:686] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 LEADER]: Becoming Leader. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Running, Role: LEADER
I20260926 07:34:09.935675 31504 consensus_queue.cc:237] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:09.939841 30986 catalog_manager.cc:5495] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "98111d5a00bf4364a9d14560e5ea6a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } health_report { overall_health: HEALTHY } } }
I20260926 07:34:09.992831 31495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.22.221.67:35345)
0926 07:34:09.987967 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.987972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.987990 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:09.988182 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.988186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.988268 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.988389 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:09.988683 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.989371 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.990707 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.991186 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.991303 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.991697 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:09.992255 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:09.992378 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.992464 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.992537 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:34:10.000512 31149 raft_consensus.cc:1261] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Refusing update from remote peer 98111d5a00bf4364a9d14560e5ea6a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:10.000550 31283 raft_consensus.cc:1261] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Refusing update from remote peer 98111d5a00bf4364a9d14560e5ea6a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:10.001452 31504 consensus_queue.cc:1035] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.001739 31506 consensus_queue.cc:1035] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.016083 31509 mvcc.cc:205] Tried to move back new op lower bound from 7333511372793036800 to 7333511372540321792. Current Snapshot: MvccSnapshot[applied={T|T < 7333511372793036800}]
I20260926 07:34:10.017006 31511 mvcc.cc:205] Tried to move back new op lower bound from 7333511372793036800 to 7333511372540321792. Current Snapshot: MvccSnapshot[applied={T|T < 7333511372793036800 or (T in {7333511372793036800})}]
I20260926 07:34:10.028268 30986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:10.028519 30986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:10.032265 30986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Sending DeleteTablet for 3 replicas of tablet 2b3482d8b32746d4b1366e53d94fb94d
I20260926 07:34:10.033414 31263 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3482d8b32746d4b1366e53d94fb94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:39310
I20260926 07:34:10.033409 31124 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3482d8b32746d4b1366e53d94fb94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260926 07:34:10.033767 31402 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3482d8b32746d4b1366e53d94fb94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:55628
I20260926 07:34:10.036084 30986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:10.036700 30986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:10.045090 31519 tablet_replica.cc:330] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: stopping tablet replica
I20260926 07:34:10.045779 31519 raft_consensus.cc:2229] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:10.046370 31262 tablet_service.cc:1467] Processing CreateTablet for tablet ccdcef0ca3a04936b6e0ab083754295a (DEFAULT_TABLE table=test-table [id=047dbbebfbd54f90bdc6db8b25581633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.046634 31124 tablet_service.cc:1467] Processing CreateTablet for tablet ccdcef0ca3a04936b6e0ab083754295a (DEFAULT_TABLE table=test-table [id=047dbbebfbd54f90bdc6db8b25581633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.046988 31262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdcef0ca3a04936b6e0ab083754295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.047190 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdcef0ca3a04936b6e0ab083754295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.052452 31401 tablet_service.cc:1467] Processing CreateTablet for tablet ccdcef0ca3a04936b6e0ab083754295a (DEFAULT_TABLE table=test-table [id=047dbbebfbd54f90bdc6db8b25581633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.053079 31401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdcef0ca3a04936b6e0ab083754295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.053112 31519 raft_consensus.cc:2258] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.054292 31496 tablet_bootstrap.cc:492] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Bootstrap starting.
I20260926 07:34:10.055935 31496 tablet_bootstrap.cc:654] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.061852 31518 tablet_replica.cc:330] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: stopping tablet replica
I20260926 07:34:10.062923 31518 raft_consensus.cc:2229] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:10.064430 31496 tablet_bootstrap.cc:492] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: No bootstrap required, opened a new log
I20260926 07:34:10.064631 31496 ts_tablet_manager.cc:1397] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260926 07:34:10.065081 31519 ts_tablet_manager.cc:1905] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.065526 31496 raft_consensus.cc:348] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.065853 31496 raft_consensus.cc:374] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.066002 31496 raft_consensus.cc:729] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8b37728c944a73bafb9edc1d10d766, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.066394 31496 consensus_queue.cc:260] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.069521 31498 tablet_bootstrap.cc:492] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Bootstrap starting.
I20260926 07:34:10.069797 31496 ts_tablet_manager.cc:1428] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260926 07:34:10.069918 31499 tablet_bootstrap.cc:492] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Bootstrap starting.
I20260926 07:34:10.071275 31498 tablet_bootstrap.cc:654] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.071583 31499 tablet_bootstrap.cc:654] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.072093 31518 raft_consensus.cc:2258] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.078562 31519 ts_tablet_manager.cc:1918] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:10.080374 31499 tablet_bootstrap.cc:492] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: No bootstrap required, opened a new log
I20260926 07:34:10.080534 31499 ts_tablet_manager.cc:1397] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260926 07:34:10.081270 31520 tablet_replica.cc:330] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: stopping tablet replica
I20260926 07:34:10.081391 31499 raft_consensus.cc:348] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.081600 31499 raft_consensus.cc:374] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.081697 31499 raft_consensus.cc:729] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.081993 31520 raft_consensus.cc:2229] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:10.082794 31520 raft_consensus.cc:2258] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.083412 31519 log.cc:1196] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/2b3482d8b32746d4b1366e53d94fb94d
I20260926 07:34:10.084137 31519 ts_tablet_manager.cc:1939] T 2b3482d8b32746d4b1366e53d94fb94d P ac8b37728c944a73bafb9edc1d10d766: Deleting consensus metadata
I20260926 07:34:10.085278 31520 ts_tablet_manager.cc:1905] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.086257 30972 catalog_manager.cc:4841] TS ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073): tablet 2b3482d8b32746d4b1366e53d94fb94d (table test-table [id=083b1ac2726d47499c8feac17d90656b]) successfully deleted
I20260926 07:34:10.087038 31498 tablet_bootstrap.cc:492] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: No bootstrap required, opened a new log
I20260926 07:34:10.087164 31518 ts_tablet_manager.cc:1905] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.087190 31498 ts_tablet_manager.cc:1397] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.001s
I20260926 07:34:10.087972 31498 raft_consensus.cc:348] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.088191 31498 raft_consensus.cc:374] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.088280 31498 raft_consensus.cc:729] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.088411 31499 consensus_queue.cc:260] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.088543 31498 consensus_queue.cc:260] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.089154 31498 ts_tablet_manager.cc:1428] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:34:10.089643 31499 ts_tablet_manager.cc:1428] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260926 07:34:10.094494 31520 ts_tablet_manager.cc:1918] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:10.094647 31520 log.cc:1196] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/2b3482d8b32746d4b1366e53d94fb94d
I20260926 07:34:10.095165 31520 ts_tablet_manager.cc:1939] T 2b3482d8b32746d4b1366e53d94fb94d P 98111d5a00bf4364a9d14560e5ea6a58: Deleting consensus metadata
I20260926 07:34:10.096881 30973 catalog_manager.cc:4841] TS 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345): tablet 2b3482d8b32746d4b1366e53d94fb94d (table test-table [id=083b1ac2726d47499c8feac17d90656b]) successfully deleted
I20260926 07:34:10.102123 31518 ts_tablet_manager.cc:1918] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:10.102267 31518 log.cc:1196] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/2b3482d8b32746d4b1366e53d94fb94d
I20260926 07:34:10.102766 31518 ts_tablet_manager.cc:1939] T 2b3482d8b32746d4b1366e53d94fb94d P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting consensus metadata
I20260926 07:34:10.104233 30973 catalog_manager.cc:4841] TS c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661): tablet 2b3482d8b32746d4b1366e53d94fb94d (table test-table [id=083b1ac2726d47499c8feac17d90656b]) successfully deleted
I20260926 07:34:10.226229 31504 raft_consensus.cc:482] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:10.226398 31504 raft_consensus.cc:504] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.227106 31504 leader_election.cc:288] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073), c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661)
I20260926 07:34:10.227674 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdcef0ca3a04936b6e0ab083754295a" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766" is_pre_election: true
I20260926 07:34:10.227708 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdcef0ca3a04936b6e0ab083754295a" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" is_pre_election: true
I20260926 07:34:10.227948 31149 raft_consensus.cc:2454] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 0.
I20260926 07:34:10.227936 31283 raft_consensus.cc:2454] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 0.
I20260926 07:34:10.228396 31355 leader_election.cc:302] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:10.228684 31504 raft_consensus.cc:2781] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:10.228830 31504 raft_consensus.cc:482] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:10.228950 31504 raft_consensus.cc:3037] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.232059 31504 raft_consensus.cc:504] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.232663 31504 leader_election.cc:288] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 election: Requested vote from peers ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073), c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661)
I20260926 07:34:10.233158 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdcef0ca3a04936b6e0ab083754295a" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766"
I20260926 07:34:10.233150 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdcef0ca3a04936b6e0ab083754295a" candidate_uuid: "98111d5a00bf4364a9d14560e5ea6a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc"
I20260926 07:34:10.233347 31283 raft_consensus.cc:3037] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.233350 31149 raft_consensus.cc:3037] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.235579 31149 raft_consensus.cc:2454] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 1.
I20260926 07:34:10.236052 31354 leader_election.cc:302] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, ac8b37728c944a73bafb9edc1d10d766; no voters:
I20260926 07:34:10.236356 31504 raft_consensus.cc:2781] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:10.236356 31283 raft_consensus.cc:2454] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98111d5a00bf4364a9d14560e5ea6a58 in term 1.
I20260926 07:34:10.236536 31504 raft_consensus.cc:686] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 LEADER]: Becoming Leader. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Running, Role: LEADER
I20260926 07:34:10.237063 31504 consensus_queue.cc:237] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.239739 30985 catalog_manager.cc:5495] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "98111d5a00bf4364a9d14560e5ea6a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } health_report { overall_health: HEALTHY } } }
Sep 26 07:34:10 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408047, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.126@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:10.303336 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.289444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.22.221.126:35765)
0926 07:34:10.290377 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.290383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.290401 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:10.290450 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.290453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.290510 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.290668 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.291899 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.292799 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.294006 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.294402 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.294543 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.298008 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:10.298013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:10.298016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:10.298042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:10.300640 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.301286 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.301292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.301299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.301418 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.301781 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.301788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.301791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.302029 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:10.302196 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.302545 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.302915 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":796,"thread_start_us":632,"threads_started":1}
I20260926 07:34:10.303344 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.289363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35765 (local address 127.0.0.1:43692)
0926 07:34:10.290065 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.290092 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.290259 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.290710 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.290715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.291506 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:10.291721 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.291730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.292943 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.292948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.293871 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.293882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.294557 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.294562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.294741 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.295483 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:10.295512 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:10.297820 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:10.300763 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.300777 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.300786 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.301173 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.301527 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.301533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.301537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.301679 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.302153 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:10.302161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:10.302314 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.302474 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.302798 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":395,"threads_started":1}
I20260926 07:34:10.308703 30985 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:43692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:34:10.309726 30985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:10.316349 31124 tablet_service.cc:1467] Processing CreateTablet for tablet 772d6d649dfa40e6aeb670185b5aa507 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28bf9553c3524480865fe5ed8630eee5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:34:10.316402 31263 tablet_service.cc:1467] Processing CreateTablet for tablet 772d6d649dfa40e6aeb670185b5aa507 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28bf9553c3524480865fe5ed8630eee5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:34:10.316558 31402 tablet_service.cc:1467] Processing CreateTablet for tablet 772d6d649dfa40e6aeb670185b5aa507 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28bf9553c3524480865fe5ed8630eee5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:34:10.316965 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772d6d649dfa40e6aeb670185b5aa507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.316974 31263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772d6d649dfa40e6aeb670185b5aa507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.317065 31402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772d6d649dfa40e6aeb670185b5aa507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.324592 31499 tablet_bootstrap.cc:492] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: Bootstrap starting.
I20260926 07:34:10.324595 31496 tablet_bootstrap.cc:492] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: Bootstrap starting.
I20260926 07:34:10.324723 31498 tablet_bootstrap.cc:492] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: Bootstrap starting.
I20260926 07:34:10.326287 31496 tablet_bootstrap.cc:654] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.326296 31499 tablet_bootstrap.cc:654] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.326327 31498 tablet_bootstrap.cc:654] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.336954 31496 tablet_bootstrap.cc:492] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: No bootstrap required, opened a new log
I20260926 07:34:10.337191 31496 ts_tablet_manager.cc:1397] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260926 07:34:10.337764 31499 tablet_bootstrap.cc:492] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: No bootstrap required, opened a new log
I20260926 07:34:10.337958 31499 ts_tablet_manager.cc:1397] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260926 07:34:10.338167 31496 raft_consensus.cc:348] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.338490 31496 raft_consensus.cc:374] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.338622 31496 raft_consensus.cc:729] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8b37728c944a73bafb9edc1d10d766, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.338698 31499 raft_consensus.cc:348] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.339054 31499 raft_consensus.cc:374] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.339169 31499 raft_consensus.cc:729] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.339077 31496 consensus_queue.cc:260] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.339548 31499 consensus_queue.cc:260] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.339700 31501 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.340054 31501 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.340152 31503 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.340564 31503 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.342633 31499 ts_tablet_manager.cc:1428] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260926 07:34:10.344063 31496 ts_tablet_manager.cc:1428] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260926 07:34:10.344753 31498 tablet_bootstrap.cc:492] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: No bootstrap required, opened a new log
I20260926 07:34:10.344878 31498 ts_tablet_manager.cc:1397] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.008s
I20260926 07:34:10.345443 31498 raft_consensus.cc:348] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.345616 31498 raft_consensus.cc:374] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.345667 31498 raft_consensus.cc:729] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.345889 31498 consensus_queue.cc:260] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.346436 31498 ts_tablet_manager.cc:1428] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:34:10.346275 31504 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.346614 31504 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.558244 31503 raft_consensus.cc:482] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:10.558490 31503 raft_consensus.cc:504] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.559963 31503 leader_election.cc:288] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073), 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:10.568328 31538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.560584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:48345 (local address 127.22.221.65:44073)
0926 07:34:10.561332 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.561337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.561357 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:10.561433 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.561438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.561533 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.561654 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.562216 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.563378 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.566074 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.567296 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.567461 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.567544 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.567828 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.567976 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":513,"threads_started":1}
I20260926 07:34:10.568358 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.560219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:44073 (local address 127.22.221.66:48345)
0926 07:34:10.560761 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.560785 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.560921 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.561666 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.561670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.561695 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.562033 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.562050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.563545 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.563551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.565906 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.565921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.567461 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.567469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.567668 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.567694 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.567773 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.567929 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":33,"thread_start_us":967,"threads_started":1}
I20260926 07:34:10.568552 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.560252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:35345 (local address 127.22.221.66:36933)
0926 07:34:10.561341 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.561358 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.561474 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.561850 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.561854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.561876 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.562257 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.562267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.563545 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.563551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.565911 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.565921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.567682 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.567686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.567873 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.567905 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.568003 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.568145 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":114,"thread_start_us":284,"threads_started":1}
I20260926 07:34:10.569031 31537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.560351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:36933 (local address 127.22.221.67:35345)
0926 07:34:10.560998 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.561003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.561459 (+ 456us) server_negotiation.cc:407] Connection header received
0926 07:34:10.561609 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.561613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.561669 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.561781 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.562394 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.563407 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.566074 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.567312 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.567460 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.567544 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.568062 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.568205 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":427,"threads_started":1}
I20260926 07:34:10.569294 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772d6d649dfa40e6aeb670185b5aa507" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766" is_pre_election: true
I20260926 07:34:10.569554 31149 raft_consensus.cc:2454] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 0.
I20260926 07:34:10.569756 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772d6d649dfa40e6aeb670185b5aa507" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98111d5a00bf4364a9d14560e5ea6a58" is_pre_election: true
I20260926 07:34:10.570034 31215 leader_election.cc:302] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8b37728c944a73bafb9edc1d10d766, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:10.570103 31422 raft_consensus.cc:2454] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 0.
I20260926 07:34:10.570392 31503 raft_consensus.cc:2781] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:10.570546 31503 raft_consensus.cc:482] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:10.570654 31503 raft_consensus.cc:3037] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.572799 31503 raft_consensus.cc:504] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.573375 31503 leader_election.cc:288] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 election: Requested vote from peers ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073), 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:10.573773 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772d6d649dfa40e6aeb670185b5aa507" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766"
I20260926 07:34:10.573969 31149 raft_consensus.cc:3037] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.575899 31149 raft_consensus.cc:2454] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 1.
I20260926 07:34:10.576177 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772d6d649dfa40e6aeb670185b5aa507" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98111d5a00bf4364a9d14560e5ea6a58"
I20260926 07:34:10.576269 31215 leader_election.cc:302] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8b37728c944a73bafb9edc1d10d766, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:10.576380 31422 raft_consensus.cc:3037] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.576588 31503 raft_consensus.cc:2781] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:10.576987 31503 raft_consensus.cc:686] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 LEADER]: Becoming Leader. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Running, Role: LEADER
I20260926 07:34:10.577383 31503 consensus_queue.cc:237] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.579371 31422 raft_consensus.cc:2454] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 1.
I20260926 07:34:10.579409 31539 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: TxnStatusTablet state changed. Reason: New leader c1eac5119d1c4e6b9c3e46331708d5dc. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.579834 31539 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:34:10.580406 31541 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:34:10.580653 31541 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:34:10.581146 30985 catalog_manager.cc:5495] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc reported cstate change: term changed from 0 to 1, leader changed from <none> to c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66). New cstate: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:10.632157 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.625785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39661 (local address 127.0.0.1:39322)
0926 07:34:10.626014 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.626038 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.626189 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.626820 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.626825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.626841 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:34:10.627061 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.627080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.628340 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.628347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.629542 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.629553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.630380 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.630386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.630573 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.630624 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.631579 (+ 955us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.631599 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.631689 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.631854 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 07:34:10.632193 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.625777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.22.221.66:39661)
0926 07:34:10.626437 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.626441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.626455 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:34:10.626530 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.626536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.626612 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.626789 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:10.627287 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.628192 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.629689 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.630204 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.630363 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.630863 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.631469 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.631617 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.631734 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.631817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":341,"threads_started":1}
I20260926 07:34:10.636765 31149 raft_consensus.cc:1261] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Refusing update from remote peer c1eac5119d1c4e6b9c3e46331708d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:10.637075 31422 raft_consensus.cc:1261] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Refusing update from remote peer c1eac5119d1c4e6b9c3e46331708d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:10.637495 31539 consensus_queue.cc:1035] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.637814 31503 consensus_queue.cc:1035] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.644821 31501 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: TxnStatusTablet state changed. Reason: New leader c1eac5119d1c4e6b9c3e46331708d5dc. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.645210 31501 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.647118 31504 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: TxnStatusTablet state changed. Reason: New leader c1eac5119d1c4e6b9c3e46331708d5dc. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.647431 31504 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.648445 31545 mvcc.cc:205] Tried to move back new op lower bound from 7333511375401390080 to 7333511375168954368. Current Snapshot: MvccSnapshot[applied={T|T < 7333511375401390080}]
I20260926 07:34:10.671703 31501 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.672022 31501 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P ac8b37728c944a73bafb9edc1d10d766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.674654 31506 consensus_queue.cc:1035] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.675386 31504 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.675647 31504 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P 98111d5a00bf4364a9d14560e5ea6a58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:34:10.685777 31539 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.686043 31539 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:34:10.686276 31539 tablet_replica.cc:439] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } } }
I20260926 07:34:10.686614 31539 tablet_replica.cc:442] T 772d6d649dfa40e6aeb670185b5aa507 P c1eac5119d1c4e6b9c3e46331708d5dc: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:34:10.691994 31506 consensus_queue.cc:1035] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.731808 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.725731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39661 (local address 127.22.221.67:44239)
0926 07:34:10.726414 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.726431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.726555 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.726884 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.726887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.726896 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:34:10.727122 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.727129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.728577 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.728581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.729746 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.729758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.730734 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.730739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.730862 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.730915 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.731354 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.731363 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.731428 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.731531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":372,"threads_started":1}
I20260926 07:34:10.731842 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.725815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:44239 (local address 127.22.221.66:39661)
0926 07:34:10.726011 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.726017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.726515 (+ 498us) server_negotiation.cc:407] Connection header received
0926 07:34:10.726680 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.726687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.726769 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.726868 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:10.727280 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.728436 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.729882 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.730573 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.730698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.731087 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.731216 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.731342 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.731473 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.731549 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:34:10.770556 31537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.763171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:53421 (local address 127.22.221.67:35345)
0926 07:34:10.763409 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.763417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.764083 (+ 666us) server_negotiation.cc:407] Connection header received
0926 07:34:10.764225 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.764231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.764337 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.764425 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:10.765271 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.766355 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.768607 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.769231 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.769340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.769962 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.770063 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.770142 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.770296 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.770357 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:34:10.770840 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.763150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:35345 (local address 127.22.221.66:53421)
0926 07:34:10.763940 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.763961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.764108 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.764585 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.764590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.764613 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:34:10.765037 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.765049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.766527 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.766532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.768450 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.768465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.769552 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.769558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.769725 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.769791 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.770174 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.770185 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.770278 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.770422 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":465,"threads_started":1}
I20260926 07:34:10.804916 30986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:10.805192 30986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:10.808451 30986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Sending DeleteTablet for 3 replicas of tablet ccdcef0ca3a04936b6e0ab083754295a
I20260926 07:34:10.809284 31124 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdcef0ca3a04936b6e0ab083754295a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260926 07:34:10.809291 31402 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdcef0ca3a04936b6e0ab083754295a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:55628
I20260926 07:34:10.809731 31263 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdcef0ca3a04936b6e0ab083754295a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:39310
I20260926 07:34:10.810258 31564 tablet_replica.cc:330] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: stopping tablet replica
I20260926 07:34:10.810496 31563 tablet_replica.cc:330] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: stopping tablet replica
I20260926 07:34:10.810732 31564 raft_consensus.cc:2229] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:10.811439 31564 raft_consensus.cc:2258] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.811009 31563 raft_consensus.cc:2229] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:10.811827 31563 raft_consensus.cc:2258] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.812950 31563 ts_tablet_manager.cc:1905] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.815128 31564 ts_tablet_manager.cc:1905] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.820160 31563 ts_tablet_manager.cc:1918] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:34:10.820369 31563 log.cc:1196] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/ccdcef0ca3a04936b6e0ab083754295a
I20260926 07:34:10.820871 31563 ts_tablet_manager.cc:1939] T ccdcef0ca3a04936b6e0ab083754295a P ac8b37728c944a73bafb9edc1d10d766: Deleting consensus metadata
I20260926 07:34:10.821455 31565 tablet_replica.cc:330] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: stopping tablet replica
I20260926 07:34:10.821672 31565 raft_consensus.cc:2229] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:10.821934 31565 raft_consensus.cc:2258] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.822148 30972 catalog_manager.cc:4841] TS ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073): tablet ccdcef0ca3a04936b6e0ab083754295a (table test-table [id=047dbbebfbd54f90bdc6db8b25581633]) successfully deleted
I20260926 07:34:10.822571 31564 ts_tablet_manager.cc:1918] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:34:10.822714 31564 log.cc:1196] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/ccdcef0ca3a04936b6e0ab083754295a
I20260926 07:34:10.823144 31565 ts_tablet_manager.cc:1905] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.823175 31564 ts_tablet_manager.cc:1939] T ccdcef0ca3a04936b6e0ab083754295a P 98111d5a00bf4364a9d14560e5ea6a58: Deleting consensus metadata
I20260926 07:34:10.824326 30973 catalog_manager.cc:4841] TS 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345): tablet ccdcef0ca3a04936b6e0ab083754295a (table test-table [id=047dbbebfbd54f90bdc6db8b25581633]) successfully deleted
I20260926 07:34:10.828475 31565 ts_tablet_manager.cc:1918] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:34:10.828580 31565 log.cc:1196] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/ccdcef0ca3a04936b6e0ab083754295a
I20260926 07:34:10.829042 31565 ts_tablet_manager.cc:1939] T ccdcef0ca3a04936b6e0ab083754295a P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting consensus metadata
I20260926 07:34:10.830026 30973 catalog_manager.cc:4841] TS c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661): tablet ccdcef0ca3a04936b6e0ab083754295a (table test-table [id=047dbbebfbd54f90bdc6db8b25581633]) successfully deleted
Sep 26 07:34:10 dist-test-slave-qlq8 krb5kdc[30942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:34:10 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:10 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:34:10.871987 31572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:10.861075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.22.221.126:35765)
0926 07:34:10.861695 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.861702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.861919 (+ 217us) server_negotiation.cc:407] Connection header received
0926 07:34:10.862151 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.862155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.862219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.862331 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.863675 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.864562 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.865816 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.866241 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.866373 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.871567 (+ 5194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43694: BlockingRecv error: recv got EOF from 127.0.0.1:43694 (error 108)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":280,"threads_started":1}
Sep 26 07:34:10 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM
I20260926 07:34:10.891013 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.875256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.22.221.126:35765)
0926 07:34:10.875468 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.875475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.875710 (+ 235us) server_negotiation.cc:407] Connection header received
0926 07:34:10.875820 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.875826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.875911 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.876026 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.877172 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.878404 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.879677 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.880384 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.880518 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.884487 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:10.884494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:10.884499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:10.884541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:10.888241 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.888901 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.888907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.888912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.889077 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.889436 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.889441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.889446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.889821 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:10.890103 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.890311 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.890624 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:34:10.894726 30986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:10.895413 30986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:10.902541 31263 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe465c79bae4ec49746ae708db6a95d (DEFAULT_TABLE table=test-table [id=29200b8c877640498540b5e22244437c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.902567 31402 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe465c79bae4ec49746ae708db6a95d (DEFAULT_TABLE table=test-table [id=29200b8c877640498540b5e22244437c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.902762 31124 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe465c79bae4ec49746ae708db6a95d (DEFAULT_TABLE table=test-table [id=29200b8c877640498540b5e22244437c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.902985 31263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe465c79bae4ec49746ae708db6a95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.903002 31402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe465c79bae4ec49746ae708db6a95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.903254 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe465c79bae4ec49746ae708db6a95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.918520 31580 tablet_bootstrap.cc:492] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Bootstrap starting.
I20260926 07:34:10.919147 31581 tablet_bootstrap.cc:492] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Bootstrap starting.
I20260926 07:34:10.919281 31579 tablet_bootstrap.cc:492] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Bootstrap starting.
I20260926 07:34:10.921555 31579 tablet_bootstrap.cc:654] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.921984 31580 tablet_bootstrap.cc:654] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.923803 31581 tablet_bootstrap.cc:654] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.924404 31579 tablet_bootstrap.cc:492] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: No bootstrap required, opened a new log
I20260926 07:34:10.924571 31579 ts_tablet_manager.cc:1397] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 07:34:10.925511 31579 raft_consensus.cc:348] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.925800 31579 raft_consensus.cc:374] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.925904 31579 raft_consensus.cc:729] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.926376 31581 tablet_bootstrap.cc:492] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: No bootstrap required, opened a new log
I20260926 07:34:10.926201 31579 consensus_queue.cc:260] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.926525 31581 ts_tablet_manager.cc:1397] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 07:34:10.927126 31581 raft_consensus.cc:348] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.927417 31581 raft_consensus.cc:374] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.927516 31581 raft_consensus.cc:729] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.927549 31579 ts_tablet_manager.cc:1428] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:34:10.928789 31580 tablet_bootstrap.cc:492] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: No bootstrap required, opened a new log
I20260926 07:34:10.928758 31581 consensus_queue.cc:260] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.929075 31580 ts_tablet_manager.cc:1397] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 07:34:10.930078 31581 ts_tablet_manager.cc:1428] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:10.929937 31580 raft_consensus.cc:348] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.930300 31580 raft_consensus.cc:374] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.930404 31580 raft_consensus.cc:729] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8b37728c944a73bafb9edc1d10d766, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.930616 31580 consensus_queue.cc:260] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:10.931528 31580 ts_tablet_manager.cc:1428] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:34:11.035866 31503 raft_consensus.cc:482] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:11.036092 31503 raft_consensus.cc:504] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.036978 31503 leader_election.cc:288] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073), 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:11.037470 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe465c79bae4ec49746ae708db6a95d" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98111d5a00bf4364a9d14560e5ea6a58" is_pre_election: true
I20260926 07:34:11.037518 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe465c79bae4ec49746ae708db6a95d" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766" is_pre_election: true
I20260926 07:34:11.037762 31421 raft_consensus.cc:2454] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 0.
I20260926 07:34:11.037792 31149 raft_consensus.cc:2454] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 0.
I20260926 07:34:11.038269 31216 leader_election.cc:302] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:11.038576 31503 raft_consensus.cc:2781] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:11.038730 31503 raft_consensus.cc:482] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:11.038858 31503 raft_consensus.cc:3037] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.041224 31503 raft_consensus.cc:504] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.041817 31503 leader_election.cc:288] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 election: Requested vote from peers ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073), 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:11.042330 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe465c79bae4ec49746ae708db6a95d" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98111d5a00bf4364a9d14560e5ea6a58"
I20260926 07:34:11.042321 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe465c79bae4ec49746ae708db6a95d" candidate_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8b37728c944a73bafb9edc1d10d766"
I20260926 07:34:11.042553 31421 raft_consensus.cc:3037] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.042567 31149 raft_consensus.cc:3037] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.044646 31421 raft_consensus.cc:2454] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 1.
I20260926 07:34:11.045069 31216 leader_election.cc:302] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:11.045351 31503 raft_consensus.cc:2781] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:11.045511 31503 raft_consensus.cc:686] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 LEADER]: Becoming Leader. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Running, Role: LEADER
I20260926 07:34:11.045778 31503 consensus_queue.cc:237] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.046093 31149 raft_consensus.cc:2454] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1eac5119d1c4e6b9c3e46331708d5dc in term 1.
I20260926 07:34:11.049209 30987 catalog_manager.cc:5495] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc reported cstate change: term changed from 0 to 1, leader changed from <none> to c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66). New cstate: current_term: 1 leader_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:11.072057 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.067211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.22.221.66:39661)
0926 07:34:11.067404 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.067411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.067431 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:11.067513 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.067518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.067602 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.067705 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:11.068032 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.069025 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.070349 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.070739 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.070870 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.071355 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:11.071487 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:11.071593 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.071766 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.071825 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:11.075023 31149 raft_consensus.cc:1261] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Refusing update from remote peer c1eac5119d1c4e6b9c3e46331708d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:11.075030 31421 raft_consensus.cc:1261] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Refusing update from remote peer c1eac5119d1c4e6b9c3e46331708d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:11.075599 31539 consensus_queue.cc:1035] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:11.075839 31503 consensus_queue.cc:1035] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:11.088838 30986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:11.089203 30986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:11.092291 30986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Sending DeleteTablet for 3 replicas of tablet 9fe465c79bae4ec49746ae708db6a95d
I20260926 07:34:11.093026 31124 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe465c79bae4ec49746ae708db6a95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:11 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260926 07:34:11.093185 31402 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe465c79bae4ec49746ae708db6a95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:11 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:55628
I20260926 07:34:11.093298 31563 tablet_replica.cc:330] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: stopping tablet replica
I20260926 07:34:11.093464 31564 tablet_replica.cc:330] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: stopping tablet replica
I20260926 07:34:11.093497 31563 raft_consensus.cc:2229] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:11.093678 31564 raft_consensus.cc:2229] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:11.093693 31563 raft_consensus.cc:2258] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:11.093902 31564 raft_consensus.cc:2258] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:11.094192 31263 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe465c79bae4ec49746ae708db6a95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:11 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:39310
I20260926 07:34:11.094468 31565 tablet_replica.cc:330] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: stopping tablet replica
I20260926 07:34:11.094653 31565 raft_consensus.cc:2229] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:11.094964 31564 ts_tablet_manager.cc:1905] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:11.095118 31565 raft_consensus.cc:2258] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:11.096062 31563 ts_tablet_manager.cc:1905] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:11.099510 31564 ts_tablet_manager.cc:1918] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:11.099676 31564 log.cc:1196] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/9fe465c79bae4ec49746ae708db6a95d
I20260926 07:34:11.100353 30986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:34:11.100761 31565 ts_tablet_manager.cc:1905] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:34:11.101001 30986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:11.102370 31563 ts_tablet_manager.cc:1918] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:11.102505 31563 log.cc:1196] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/9fe465c79bae4ec49746ae708db6a95d
I20260926 07:34:11.102893 31563 ts_tablet_manager.cc:1939] T 9fe465c79bae4ec49746ae708db6a95d P ac8b37728c944a73bafb9edc1d10d766: Deleting consensus metadata
I20260926 07:34:11.103747 31564 ts_tablet_manager.cc:1939] T 9fe465c79bae4ec49746ae708db6a95d P 98111d5a00bf4364a9d14560e5ea6a58: Deleting consensus metadata
I20260926 07:34:11.105515 30973 catalog_manager.cc:4841] TS 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345): tablet 9fe465c79bae4ec49746ae708db6a95d (table test-table [id=29200b8c877640498540b5e22244437c]) successfully deleted
I20260926 07:34:11.108196 31565 ts_tablet_manager.cc:1918] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:11.108425 31565 log.cc:1196] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/9fe465c79bae4ec49746ae708db6a95d
I20260926 07:34:11.108522 31124 tablet_service.cc:1467] Processing CreateTablet for tablet 266b75b77459445884328696b5831438 (DEFAULT_TABLE table=test-table [id=7bc9df04c1434842b78fda3f15655225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:11.108920 31565 ts_tablet_manager.cc:1939] T 9fe465c79bae4ec49746ae708db6a95d P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting consensus metadata
I20260926 07:34:11.108949 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266b75b77459445884328696b5831438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.111505 30972 catalog_manager.cc:4841] TS ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073): tablet 9fe465c79bae4ec49746ae708db6a95d (table test-table [id=29200b8c877640498540b5e22244437c]) successfully deleted
I20260926 07:34:11.111676 30973 catalog_manager.cc:4841] TS c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661): tablet 9fe465c79bae4ec49746ae708db6a95d (table test-table [id=29200b8c877640498540b5e22244437c]) successfully deleted
I20260926 07:34:11.112303 31402 tablet_service.cc:1467] Processing CreateTablet for tablet 266b75b77459445884328696b5831438 (DEFAULT_TABLE table=test-table [id=7bc9df04c1434842b78fda3f15655225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:11.112668 31402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266b75b77459445884328696b5831438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.113831 31263 tablet_service.cc:1467] Processing CreateTablet for tablet 266b75b77459445884328696b5831438 (DEFAULT_TABLE table=test-table [id=7bc9df04c1434842b78fda3f15655225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:11.114241 31263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266b75b77459445884328696b5831438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.117406 31581 tablet_bootstrap.cc:492] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Bootstrap starting.
I20260926 07:34:11.118953 31581 tablet_bootstrap.cc:654] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.120666 31581 tablet_bootstrap.cc:492] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: No bootstrap required, opened a new log
I20260926 07:34:11.120846 31581 ts_tablet_manager.cc:1397] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:34:11.121204 31579 tablet_bootstrap.cc:492] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Bootstrap starting.
I20260926 07:34:11.121652 31581 raft_consensus.cc:348] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.121912 31581 raft_consensus.cc:374] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.121994 31581 raft_consensus.cc:729] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98111d5a00bf4364a9d14560e5ea6a58, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.122263 31581 consensus_queue.cc:260] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.122962 31579 tablet_bootstrap.cc:654] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.123591 31581 ts_tablet_manager.cc:1428] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:11.126178 31580 tablet_bootstrap.cc:492] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Bootstrap starting.
I20260926 07:34:11.128252 31580 tablet_bootstrap.cc:654] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.128975 31579 tablet_bootstrap.cc:492] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: No bootstrap required, opened a new log
I20260926 07:34:11.129186 31579 ts_tablet_manager.cc:1397] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260926 07:34:11.129983 31579 raft_consensus.cc:348] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.130311 31580 tablet_bootstrap.cc:492] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: No bootstrap required, opened a new log
I20260926 07:34:11.130313 31579 raft_consensus.cc:374] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.130542 31580 ts_tablet_manager.cc:1397] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260926 07:34:11.130570 31579 raft_consensus.cc:729] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1eac5119d1c4e6b9c3e46331708d5dc, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.131191 31579 consensus_queue.cc:260] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.131295 31580 raft_consensus.cc:348] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.131769 31580 raft_consensus.cc:374] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.131814 31579 ts_tablet_manager.cc:1428] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:11.131848 31580 raft_consensus.cc:729] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8b37728c944a73bafb9edc1d10d766, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.132091 31580 consensus_queue.cc:260] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.133152 31580 ts_tablet_manager.cc:1428] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:34:11.148375 31501 raft_consensus.cc:482] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:11.148583 31501 raft_consensus.cc:504] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.150414 31501 leader_election.cc:288] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661), 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:11.158303 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.150582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:47495 (local address 127.22.221.66:39661)
0926 07:34:11.150781 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.150787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.151264 (+ 477us) server_negotiation.cc:407] Connection header received
0926 07:34:11.151572 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.151581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.151676 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.151774 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.152910 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.154117 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.156205 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.157353 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.157534 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.157612 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.157905 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.158051 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:11.158371 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.150553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39661 (local address 127.22.221.65:47495)
0926 07:34:11.151153 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.151173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.151369 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.151936 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.151939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.151961 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.152247 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.152262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.154462 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.154467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.155995 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.156008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.157509 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.157517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.157694 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.157724 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.157843 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.157977 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":3,"thread_start_us":308,"threads_started":1}
I20260926 07:34:11.158946 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266b75b77459445884328696b5831438" candidate_uuid: "ac8b37728c944a73bafb9edc1d10d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" is_pre_election: true
I20260926 07:34:11.159248 31283 raft_consensus.cc:2454] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac8b37728c944a73bafb9edc1d10d766 in term 0.
I20260926 07:34:11.159484 31537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.151393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:57081 (local address 127.22.221.67:35345)
0926 07:34:11.151584 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.151590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.152158 (+ 568us) server_negotiation.cc:407] Connection header received
0926 07:34:11.152325 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.152333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.152419 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.152521 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.153195 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.154216 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.156728 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.157968 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.158133 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.158231 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.159045 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.159173 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:11.159837 31076 leader_election.cc:302] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8b37728c944a73bafb9edc1d10d766, c1eac5119d1c4e6b9c3e46331708d5dc; no voters:
I20260926 07:34:11.160250 31501 raft_consensus.cc:2781] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:11.160254 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.151255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:35345 (local address 127.22.221.65:57081)
0926 07:34:11.152033 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.152050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.152180 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.152538 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.152543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.152569 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.152954 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.152966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.154349 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.154355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.156528 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.156541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.158671 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.158680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.158852 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.158897 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.158994 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.159199 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":25,"thread_start_us":240,"threads_started":1}
I20260926 07:34:11.160413 31501 raft_consensus.cc:482] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:11.160519 31501 raft_consensus.cc:3037] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.160919 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266b75b77459445884328696b5831438" candidate_uuid: "ac8b37728c944a73bafb9edc1d10d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98111d5a00bf4364a9d14560e5ea6a58" is_pre_election: true
I20260926 07:34:11.161161 31421 raft_consensus.cc:2454] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac8b37728c944a73bafb9edc1d10d766 in term 0.
I20260926 07:34:11.163534 31501 raft_consensus.cc:504] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.164212 31501 leader_election.cc:288] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [CANDIDATE]: Term 1 election: Requested vote from peers c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661), 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345)
I20260926 07:34:11.164691 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266b75b77459445884328696b5831438" candidate_uuid: "ac8b37728c944a73bafb9edc1d10d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98111d5a00bf4364a9d14560e5ea6a58"
I20260926 07:34:11.164882 31421 raft_consensus.cc:3037] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.167735 31421 raft_consensus.cc:2454] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac8b37728c944a73bafb9edc1d10d766 in term 1.
I20260926 07:34:11.168087 31076 leader_election.cc:302] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98111d5a00bf4364a9d14560e5ea6a58, ac8b37728c944a73bafb9edc1d10d766; no voters:
I20260926 07:34:11.171479 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266b75b77459445884328696b5831438" candidate_uuid: "ac8b37728c944a73bafb9edc1d10d766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc"
I20260926 07:34:11.171729 31283 raft_consensus.cc:3037] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.171981 31501 raft_consensus.cc:2781] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:11.174880 31283 raft_consensus.cc:2454] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac8b37728c944a73bafb9edc1d10d766 in term 1.
I20260926 07:34:11.176502 31501 raft_consensus.cc:686] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 1 LEADER]: Becoming Leader. State: Replica: ac8b37728c944a73bafb9edc1d10d766, State: Running, Role: LEADER
I20260926 07:34:11.176954 31501 consensus_queue.cc:237] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } }
I20260926 07:34:11.180433 30986 catalog_manager.cc:5495] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "ac8b37728c944a73bafb9edc1d10d766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8b37728c944a73bafb9edc1d10d766" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:11.209313 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.202150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.22.221.65:44073)
0926 07:34:11.202703 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.202709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.202729 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:11.202816 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.202821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.202917 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.203132 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:11.203655 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.204827 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.206808 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.207280 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.207472 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.207995 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:11.208622 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:11.208759 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.208894 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.208956 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
I20260926 07:34:11.224701 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.219115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:49183 (local address 127.22.221.66:39661)
0926 07:34:11.219311 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.219315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.219876 (+ 561us) server_negotiation.cc:407] Connection header received
0926 07:34:11.220108 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.220113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.220207 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.220304 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:11.220751 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.221449 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.222766 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.223450 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.223604 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.224001 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:11.224134 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:11.224246 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.224417 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.224483 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:11.224933 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.219146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:39661 (local address 127.22.221.65:49183)
0926 07:34:11.219772 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.219787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.219942 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.220328 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.220332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.220344 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:34:11.220614 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.220621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.221605 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.221615 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.222628 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.222640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.223635 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.223641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.223774 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.223820 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:34:11.224276 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:34:11.224295 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.224392 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.224532 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":323,"threads_started":1}
I20260926 07:34:11.232758 31422 raft_consensus.cc:1261] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Refusing update from remote peer ac8b37728c944a73bafb9edc1d10d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:11.232882 31283 raft_consensus.cc:1261] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Refusing update from remote peer ac8b37728c944a73bafb9edc1d10d766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:11.233470 31594 consensus_queue.cc:1035] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98111d5a00bf4364a9d14560e5ea6a58" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:11.233882 31501 consensus_queue.cc:1035] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1eac5119d1c4e6b9c3e46331708d5dc" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:11.262880 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.256709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:60119 (local address 127.22.221.65:44073)
0926 07:34:11.256859 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.256865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.256883 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:11.256939 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.256944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.257029 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.257121 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:11.257847 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.258863 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.260804 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.261484 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.261612 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.262088 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:11.262254 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:11.262362 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.262514 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.262587 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:34:11.262917 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.256181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:44073 (local address 127.22.221.66:60119)
0926 07:34:11.256373 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.256393 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.256496 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.257482 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.257487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.257500 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:34:11.257719 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.257732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.259001 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.259014 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.260649 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.260663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.261655 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.261661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.261864 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.261913 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:34:11.262363 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:34:11.262376 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.262489 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.262621 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:34:11.291627 30986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:11.291875 30986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:11.294823 30986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5885682c4e6140b0be8bbf3fd1dbcd26: Sending DeleteTablet for 3 replicas of tablet 266b75b77459445884328696b5831438
I20260926 07:34:11.295578 31124 tablet_service.cc:1514] Processing DeleteTablet for tablet 266b75b77459445884328696b5831438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:11 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260926 07:34:11.295892 31563 tablet_replica.cc:330] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: stopping tablet replica
I20260926 07:34:11.295907 31402 tablet_service.cc:1514] Processing DeleteTablet for tablet 266b75b77459445884328696b5831438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:11 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:55628
I20260926 07:34:11.296079 31563 raft_consensus.cc:2229] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:11.296183 31564 tablet_replica.cc:330] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: stopping tablet replica
I20260926 07:34:11.296356 31564 raft_consensus.cc:2229] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:11.296566 31564 raft_consensus.cc:2258] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:11.296576 31563 raft_consensus.cc:2258] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:11.297936 31563 ts_tablet_manager.cc:1905] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:11.298099 31564 ts_tablet_manager.cc:1905] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:11.299386 31263 tablet_service.cc:1514] Processing DeleteTablet for tablet 266b75b77459445884328696b5831438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:11 UTC) from {username='oryx', principal='oryx/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:39310
I20260926 07:34:11.299654 31565 tablet_replica.cc:330] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: stopping tablet replica
I20260926 07:34:11.299823 31565 raft_consensus.cc:2229] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:11.300020 31565 raft_consensus.cc:2258] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:11.301309 31565 ts_tablet_manager.cc:1905] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:11.305246 31564 ts_tablet_manager.cc:1918] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:34:11.305361 31564 log.cc:1196] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/266b75b77459445884328696b5831438
I20260926 07:34:11.305753 31564 ts_tablet_manager.cc:1939] T 266b75b77459445884328696b5831438 P 98111d5a00bf4364a9d14560e5ea6a58: Deleting consensus metadata
I20260926 07:34:11.306793 30973 catalog_manager.cc:4841] TS 98111d5a00bf4364a9d14560e5ea6a58 (127.22.221.67:35345): tablet 266b75b77459445884328696b5831438 (table test-table [id=7bc9df04c1434842b78fda3f15655225]) successfully deleted
I20260926 07:34:11.310928 31563 ts_tablet_manager.cc:1918] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:34:11.311090 31563 log.cc:1196] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/266b75b77459445884328696b5831438
I20260926 07:34:11.313156 31565 ts_tablet_manager.cc:1918] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:34:11.313387 31565 log.cc:1196] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/266b75b77459445884328696b5831438
I20260926 07:34:11.313934 31565 ts_tablet_manager.cc:1939] T 266b75b77459445884328696b5831438 P c1eac5119d1c4e6b9c3e46331708d5dc: Deleting consensus metadata
I20260926 07:34:11.314524 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 31058
I20260926 07:34:11.315177 30973 catalog_manager.cc:4841] TS c1eac5119d1c4e6b9c3e46331708d5dc (127.22.221.66:39661): tablet 266b75b77459445884328696b5831438 (table test-table [id=7bc9df04c1434842b78fda3f15655225]) successfully deleted
I20260926 07:34:11.316025 31563 ts_tablet_manager.cc:1939] T 266b75b77459445884328696b5831438 P ac8b37728c944a73bafb9edc1d10d766: Deleting consensus metadata
I20260926 07:34:11.317430 30972 catalog_manager.cc:4841] TS ac8b37728c944a73bafb9edc1d10d766 (127.22.221.65:44073): tablet 266b75b77459445884328696b5831438 (table test-table [id=7bc9df04c1434842b78fda3f15655225]) successfully deleted
Sep 26 07:34:11 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.65@KRBTEST.COM
I20260926 07:34:11.332659 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.315691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.22.221.65:44073)
0926 07:34:11.315973 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.315981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.317724 (+ 1743us) server_negotiation.cc:407] Connection header received
0926 07:34:11.317994 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.318005 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.318068 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.318184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.319364 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.320103 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.321341 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.321770 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.321918 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.326156 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.326176 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.326190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:11.326219 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.329571 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:11.330304 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:11.330310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:11.330312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:11.330476 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:11.330865 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:11.330870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:11.330873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:11.331361 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.331671 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.331997 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.332222 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260926 07:34:11.333336 31189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:11.446280 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 31058
I20260926 07:34:11.465749 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 31198
Sep 26 07:34:11 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.66@KRBTEST.COM
I20260926 07:34:11.481623 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.466674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.22.221.66:39661)
0926 07:34:11.466872 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.466877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.466899 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:11.467021 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.467026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.467129 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.467275 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.468450 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.469263 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.470573 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.471049 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.471193 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.475580 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.475593 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.475607 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:11.475648 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.478727 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:11.479467 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:11.479476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:11.479480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:11.479626 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:11.480002 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:11.480008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:11.480012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:11.480367 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.480686 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.481037 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.481302 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:34:11.482287 31328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:11.636971 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 31198
I20260926 07:34:11.661573 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 31337
Sep 26 07:34:11 dist-test-slave-qlq8 krb5kdc[30942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.221.67@KRBTEST.COM
I20260926 07:34:11.678403 31608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.662363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.22.221.67:35345)
0926 07:34:11.663056 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.663061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.663079 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:11.663134 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.663139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.663214 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.663342 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.664645 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.665738 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.666919 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.667584 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.667698 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.671960 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.671976 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.671996 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:11.672038 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.675767 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:11.676478 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:11.676487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:11.676492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:11.676630 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:11.676948 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:11.676955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:11.676962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:11.677360 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.677630 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.677798 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.677999 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":460,"threads_started":1}
I20260926 07:34:11.679132 31467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:11.804785 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 31337
I20260926 07:34:11.827049 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 30955
I20260926 07:34:11.828374 31027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:11.945256 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 30955
2026-09-26T07:34:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5619 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:12 dist-test-slave-qlq8 krb5kdc[31613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:12 dist-test-slave-qlq8 krb5kdc[31613](info): set up 2 sockets
Sep 26 07:34:12 dist-test-slave-qlq8 krb5kdc[31613](info): commencing operation
krb5kdc: starting...
W20260926 07:34:13.048476 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:34:13 dist-test-slave-qlq8 krb5kdc[31613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:34:13.254060 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46875
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:46875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:13.567627 31626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:13.567888 31626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:13.567974 31626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:13.574836 31626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:13.574922 31626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:13.574967 31626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:13.575017 31626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:13.575058 31626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:13.584379 31626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:46875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:13.585489 31626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:13.586678 31626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:13.598335 31632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:13.598363 31634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:13.598953 31631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:13.599251 31626 server_base.cc:1034] running on GCE node
I20260926 07:34:13.599974 31626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:13.601219 31626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:13.602415 31626 hybrid_clock.cc:648] HybridClock initialized: now 1790408053602374 us; error 39 us; skew 500 ppm
I20260926 07:34:13.602751 31626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T07:34:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1721 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:13 dist-test-slave-qlq8 krb5kdc[31640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:13 dist-test-slave-qlq8 krb5kdc[31640](info): set up 2 sockets
Sep 26 07:34:13 dist-test-slave-qlq8 krb5kdc[31640](info): commencing operation
krb5kdc: starting...
W20260926 07:34:14.792905 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:34:14 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:34:15.003273 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:33337
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:33337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:15.322167 31653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:15.322438 31653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:15.322527 31653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:15.329670 31653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:15.329766 31653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:15.329819 31653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:15.329878 31653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:15.329916 31653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:15.339553 31653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:33337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:33337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:15.340742 31653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:15.341928 31653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:15.353526 31659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:15.353626 31658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:15.353533 31661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:15.354368 31653 server_base.cc:1034] running on GCE node
I20260926 07:34:15.355121 31653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:15.356405 31653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:15.357604 31653 hybrid_clock.cc:648] HybridClock initialized: now 1790408055357560 us; error 29 us; skew 500 ppm
I20260926 07:34:15.357923 31653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:15 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408055, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:15.361011 31653 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:34:15.362409 31653 webserver.cc:460] Webserver started at http://127.22.221.126:44065/ using document root <none> and password file <none>
I20260926 07:34:15.363008 31653 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:15.363099 31653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:15.363420 31653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:15.366130 31653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "105370ee1a324c5bb8da68e2758b36a3"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "cdf37fbae33f34b8091772a24a94b341"
server_key_iv: "adcf2d7abe9c2e3f1460e08a5a920819"
server_key_version: "encryptionkey@0"
I20260926 07:34:15.366802 31653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "105370ee1a324c5bb8da68e2758b36a3"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "cdf37fbae33f34b8091772a24a94b341"
server_key_iv: "adcf2d7abe9c2e3f1460e08a5a920819"
server_key_version: "encryptionkey@0"
I20260926 07:34:15.371969 31653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:34:15.375268 31668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:15.376549 31653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:34:15.376729 31653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "105370ee1a324c5bb8da68e2758b36a3"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "cdf37fbae33f34b8091772a24a94b341"
server_key_iv: "adcf2d7abe9c2e3f1460e08a5a920819"
server_key_version: "encryptionkey@0"
I20260926 07:34:15.376946 31653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:15.390551 31653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:15.394691 31653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:15.394972 31653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:15.414132 31653 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:33337
I20260926 07:34:15.414152 31719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:33337 every 8 connection(s)
I20260926 07:34:15.415556 31653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:15.419046 31720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:15.422413 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 31653
I20260926 07:34:15.422616 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:34:15.423300 23413 external_mini_cluster.cc:1428] Setting key e7d95590c9151e92233d588860be996b
I20260926 07:34:15.432112 31720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Bootstrap starting.
I20260926 07:34:15.435736 31720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:15.436704 31720 log.cc:824] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:34:15 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:15.439669 31720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: No bootstrap required, opened a new log
I20260926 07:34:15.447219 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:15.425168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.22.221.126:33337)
0926 07:34:15.425997 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:15.426012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:15.426061 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:34:15.427910 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:15.427938 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:15.428466 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:15.428890 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:15.430155 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.432052 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.433477 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.434571 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.434742 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:15.438738 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:15.438749 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:15.438768 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:15.438826 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:15.442885 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:15.443600 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:15.443609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:15.443617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:15.443823 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:15.444191 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:15.444197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:15.444200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:15.444841 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:15.445154 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:15.445437 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:34:15.445874 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":347,"threads_started":1}
I20260926 07:34:15.450146 31720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:15.450443 31720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:15.450525 31720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105370ee1a324c5bb8da68e2758b36a3, State: Initialized, Role: FOLLOWER
I20260926 07:34:15.450973 31720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:15.451164 31720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:15.451339 31720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:15.451514 31720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:15.453800 31720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:15.454272 31720 leader_election.cc:302] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105370ee1a324c5bb8da68e2758b36a3; no voters:
I20260926 07:34:15.454842 31720 leader_election.cc:288] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:15.454988 31725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:15.455678 31725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 LEADER]: Becoming Leader. State: Replica: 105370ee1a324c5bb8da68e2758b36a3, State: Running, Role: LEADER
I20260926 07:34:15.456184 31725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:15.456833 31720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:15.458696 31727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105370ee1a324c5bb8da68e2758b36a3. Latest consensus state: current_term: 1 leader_uuid: "105370ee1a324c5bb8da68e2758b36a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } } }
I20260926 07:34:15.459223 31726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "105370ee1a324c5bb8da68e2758b36a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } } }
I20260926 07:34:15.459565 31726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:15.459662 31727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:15.462962 31731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:15.468164 31731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:15.475888 31731 catalog_manager.cc:1347] Generated new cluster ID: d9996d6b29404fa6b982ff92eaaa7d0c
I20260926 07:34:15.475997 31731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:15.486883 31731 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:15.488684 31731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:15.497869 31731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Generated new TSK 0
I20260926 07:34:15.498768 31731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:34:15.546650 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33337
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:15.867592 31746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:15.867864 31746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:15.868034 31746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:15.875504 31746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:15.875648 31746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:15.875810 31746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:15.885332 31746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:15.886595 31746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:15.887871 31746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:15.901146 31754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:15.901214 31751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:15.901191 31752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:15.901963 31746 server_base.cc:1034] running on GCE node
I20260926 07:34:15.902539 31746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:15.903524 31746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:15.904724 31746 hybrid_clock.cc:648] HybridClock initialized: now 1790408055904690 us; error 29 us; skew 500 ppm
I20260926 07:34:15.905061 31746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:15 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408055, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:15.908326 31746 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:34:15.909823 31746 webserver.cc:460] Webserver started at http://127.22.221.65:34795/ using document root <none> and password file <none>
I20260926 07:34:15.910506 31746 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:15.910634 31746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:15.910948 31746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:15.913837 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "eba67684f5b347b0849cbb292b555a82"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "a4d56d107ce9953885e593f2a6940083"
server_key_iv: "78fcb6636e599d7823655ede9a022063"
server_key_version: "encryptionkey@0"
I20260926 07:34:15.914695 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "eba67684f5b347b0849cbb292b555a82"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "a4d56d107ce9953885e593f2a6940083"
server_key_iv: "78fcb6636e599d7823655ede9a022063"
server_key_version: "encryptionkey@0"
I20260926 07:34:15.919687 31746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 07:34:15.922869 31761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:15.924271 31746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:15.924440 31746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "eba67684f5b347b0849cbb292b555a82"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "a4d56d107ce9953885e593f2a6940083"
server_key_iv: "78fcb6636e599d7823655ede9a022063"
server_key_version: "encryptionkey@0"
I20260926 07:34:15.924638 31746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:15.936743 31746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:15.940629 31746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:15.940908 31746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:15.941838 31746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:15.943082 31746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:15.943188 31746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:15.943359 31746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:15.943435 31746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:15.976876 31746 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:42359
I20260926 07:34:15.976922 31873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:42359 every 8 connection(s)
I20260926 07:34:15.978118 31746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:15.985561 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 31746
I20260926 07:34:15.985776 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:15.986418 23413 external_mini_cluster.cc:1428] Setting key 8eff473a56c3bf12afcfb9d88cbe2aa9
Sep 26 07:34:15 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408055, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:15.998916 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:15.980741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:36495 (local address 127.22.221.126:33337)
0926 07:34:15.981200 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:15.981204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:15.982053 (+ 849us) server_negotiation.cc:407] Connection header received
0926 07:34:15.983914 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:15.983920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:15.984010 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:15.984126 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:15.985717 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.986527 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.988318 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.988708 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.988835 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:15.992728 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:15.992743 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:15.992746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:15.992777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:15.995559 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:15.996261 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:15.996269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:15.996275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:15.996420 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:15.996828 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:15.996836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:15.996843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:15.997232 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:15.997407 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:15.998177 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 07:34:15.998431 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":245,"threads_started":1}
I20260926 07:34:15.999564 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:15.981132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:36495)
0926 07:34:15.981896 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:15.981931 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:15.983650 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:15.984351 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:15.984359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:15.984829 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:15.985555 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:15.985575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.986676 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.986682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:15.988179 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:15.988194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.988914 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.988924 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:15.989188 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:15.989876 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:15.989897 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:15.992514 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:15.995702 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:15.995708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:15.995719 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:15.996125 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:15.996535 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:15.996538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:15.996540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:15.996712 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:15.997358 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:15.997363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:15.997757 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 07:34:15.998026 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 07:34:15.998455 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":318,"threads_started":1}
I20260926 07:34:16.001153 31874 heartbeater.cc:347] Connected to a master server at 127.22.221.126:33337
I20260926 07:34:16.001480 31874 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:16.002192 31874 heartbeater.cc:511] Master 127.22.221.126:33337 requested a full tablet report, sending...
I20260926 07:34:16.004184 31685 ts_manager.cc:194] Registered new tserver with Master: eba67684f5b347b0849cbb292b555a82 (127.22.221.65:42359)
I20260926 07:34:16.005661 31685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:36495
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:34:16.034883 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33337
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:16.350253 31880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:16.350549 31880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:16.350723 31880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:16.357847 31880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:16.357982 31880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:16.358152 31880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:16.367334 31880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:16.368515 31880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:16.369778 31880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:16.382243 31885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:16.382272 31886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:16.382325 31888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:16.383137 31880 server_base.cc:1034] running on GCE node
I20260926 07:34:16.383701 31880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:16.384549 31880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:16.385809 31880 hybrid_clock.cc:648] HybridClock initialized: now 1790408056385742 us; error 56 us; skew 500 ppm
I20260926 07:34:16.386121 31880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:16 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408056, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:16.389351 31880 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:34:16.390744 31880 webserver.cc:460] Webserver started at http://127.22.221.66:36443/ using document root <none> and password file <none>
I20260926 07:34:16.391467 31880 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:16.391592 31880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:16.391884 31880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:16.394595 31880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "5efdf74edffe48848be0a6f9a10396f1"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "f7bc2d3941778dfdfeee036d12c4641b"
server_key_iv: "3ccbac1037fc19c5ee6a177e8f0d48cb"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.395401 31880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "5efdf74edffe48848be0a6f9a10396f1"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "f7bc2d3941778dfdfeee036d12c4641b"
server_key_iv: "3ccbac1037fc19c5ee6a177e8f0d48cb"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.400171 31880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:34:16.403446 31895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.404713 31880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:34:16.404879 31880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "5efdf74edffe48848be0a6f9a10396f1"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "f7bc2d3941778dfdfeee036d12c4641b"
server_key_iv: "3ccbac1037fc19c5ee6a177e8f0d48cb"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.405064 31880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:16.419322 31880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:16.424134 31880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:16.424404 31880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:16.425386 31880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:16.426900 31880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:16.426997 31880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.427122 31880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:16.427183 31880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.461067 31880 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:33253
I20260926 07:34:16.461112 32007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:33253 every 8 connection(s)
I20260926 07:34:16.462368 31880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:16.462832 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 31880
I20260926 07:34:16.463016 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:16.463728 23413 external_mini_cluster.cc:1428] Setting key dd9607136b5da7d7d4c4294738ee4e31
Sep 26 07:34:16 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408056, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
I20260926 07:34:16.493134 32010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:16.466445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:49669)
0926 07:34:16.475846 (+ 9401us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:16.475892 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:16.477647 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:16.478358 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:16.478366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:16.478863 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:16.479738 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:16.479766 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:16.481044 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:16.481049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:16.482271 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:16.482287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:16.483086 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:16.483095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:16.483403 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:16.484089 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:16.484111 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:16.486218 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:16.489456 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:16.489465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:16.489480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:16.489879 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:16.490292 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:16.490297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:16.490301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:16.490487 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:16.491054 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:16.491063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:16.491489 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 07:34:16.491774 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 07:34:16.492157 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9121,"thread_start_us":8870,"threads_started":1}
I20260926 07:34:16.494242 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:16.467122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:49669 (local address 127.22.221.126:33337)
0926 07:34:16.467335 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:16.467344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:16.475999 (+ 8655us) server_negotiation.cc:407] Connection header received
0926 07:34:16.477920 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:16.477929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:16.478023 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:16.478141 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:16.479985 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:16.480883 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:16.482432 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:16.482857 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:16.482995 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:16.486437 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:16.486444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:16.486448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:16.486499 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:16.489316 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:16.490048 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:16.490052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:16.490055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:16.490175 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:16.490606 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:16.490613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:16.490621 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:16.490856 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:16.491133 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:16.493534 (+ 2401us) server_negotiation.cc:299] Negotiation successful
0926 07:34:16.493814 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:34:16.495546 32008 heartbeater.cc:347] Connected to a master server at 127.22.221.126:33337
I20260926 07:34:16.495869 32008 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:16.496598 32008 heartbeater.cc:511] Master 127.22.221.126:33337 requested a full tablet report, sending...
I20260926 07:34:16.498111 31685 ts_manager.cc:194] Registered new tserver with Master: 5efdf74edffe48848be0a6f9a10396f1 (127.22.221.66:33253)
I20260926 07:34:16.499367 31685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:49669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:34:16.518704 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33337
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:16.834612 32013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:16.834862 32013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:16.834954 32013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:16.841943 32013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:16.842042 32013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:16.842154 32013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:16.851274 32013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:16.852396 32013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:16.853624 32013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:16.866534 32019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:16.866617 32021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:16.866796 32018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:16.867311 32013 server_base.cc:1034] running on GCE node
I20260926 07:34:16.867861 32013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:16.868727 32013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:16.869944 32013 hybrid_clock.cc:648] HybridClock initialized: now 1790408056869901 us; error 65 us; skew 500 ppm
I20260926 07:34:16.870249 32013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:16 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408056, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:16.895188 32013 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:34:16.896843 32013 webserver.cc:460] Webserver started at http://127.22.221.67:36101/ using document root <none> and password file <none>
I20260926 07:34:16.897573 32013 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:16.897719 32013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:16.898042 32013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:16.901221 32013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "233a7f4d4338421e99c766fd6b376e69"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "14f24ec9f41f969c80f1efc5f705672f"
server_key_iv: "4280301fb211b6095ba8076101558977"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.902002 32013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "233a7f4d4338421e99c766fd6b376e69"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "14f24ec9f41f969c80f1efc5f705672f"
server_key_iv: "4280301fb211b6095ba8076101558977"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.906925 32013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:34:16.910475 32028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.911857 32013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:16.912034 32013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "233a7f4d4338421e99c766fd6b376e69"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "14f24ec9f41f969c80f1efc5f705672f"
server_key_iv: "4280301fb211b6095ba8076101558977"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.912223 32013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:16.928196 32013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:16.932180 32013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:16.932461 32013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:16.933419 32013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:16.934748 32013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:16.934855 32013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.935015 32013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:16.935091 32013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.984390 32013 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:46077
I20260926 07:34:16.984436 32140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:46077 every 8 connection(s)
I20260926 07:34:16.986047 32013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:16.991007 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 32013
I20260926 07:34:16.991165 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:16.991676 23413 external_mini_cluster.cc:1428] Setting key 3ed864e3de35bcb6aadbc5efdd2f4d05
Sep 26 07:34:16 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408056, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:17.008548 31874 heartbeater.cc:503] Master 127.22.221.126:33337 was elected leader, sending a full tablet report...
I20260926 07:34:17.013891 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:16.990057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:46009 (local address 127.22.221.126:33337)
0926 07:34:16.990262 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:16.990269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:16.991174 (+ 905us) server_negotiation.cc:407] Connection header received
0926 07:34:16.994460 (+ 3286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:16.994467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:16.994551 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:16.994661 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:16.996559 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:16.998006 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.000687 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.001488 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.001640 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.006568 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:17.006576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:17.006581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:17.006629 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:17.010762 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.011825 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.011831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.011836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.011973 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.012360 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.012364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.012366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.012642 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:17.012847 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:17.013499 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 07:34:17.013657 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:34:17.014699 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:16.989522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:46009)
0926 07:34:16.990340 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:16.990382 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:16.994049 (+ 3667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:16.994916 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:16.994928 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:16.995672 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:16.996375 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:16.996398 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:16.998216 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:16.998222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:17.000503 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:17.000519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.001763 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.001771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:17.002106 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.002924 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:17.002967 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:17.006333 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:17.010882 (+ 4549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:17.010893 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:17.010911 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:17.011625 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:17.012080 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:17.012083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:17.012085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:17.012251 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:17.012757 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:17.012763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:17.013072 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 07:34:17.013291 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 07:34:17.013634 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":645,"threads_started":1}
I20260926 07:34:17.016078 32141 heartbeater.cc:347] Connected to a master server at 127.22.221.126:33337
I20260926 07:34:17.016373 32141 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:17.017066 32141 heartbeater.cc:511] Master 127.22.221.126:33337 requested a full tablet report, sending...
I20260926 07:34:17.018399 31685 ts_manager.cc:194] Registered new tserver with Master: 233a7f4d4338421e99c766fd6b376e69 (127.22.221.67:46077)
I20260926 07:34:17.019356 31685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:46009
I20260926 07:34:17.019873 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab.
Entry for principal oryx/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab.
WARNING: no policy specified for oryx/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab.
Entry for principal oryx/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab.
WARNING: no policy specified for oryx/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab.
Entry for principal oryx/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab.
I20260926 07:34:17.147922 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 31746
Sep 26 07:34:17 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:34:17.169209 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:17.149062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.22.221.65:42359)
0926 07:34:17.149841 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:17.149849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:17.149873 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:34:17.149933 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:17.149940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:17.150161 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:17.150331 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:17.151774 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.152818 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.154052 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.154636 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.154779 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.163809 (+ 9030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:17.163820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:17.163834 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:17.163872 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:17.166670 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.167376 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.167383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.167385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.167513 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.167824 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.167827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.167829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.168154 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:17.168372 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:17.168612 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:34:17.168806 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":422,"threads_started":1}
I20260926 07:34:17.170461 31869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:17.252388 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 31746
I20260926 07:34:17.268586 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 31880
Sep 26 07:34:17 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:34:17.283847 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:17.269502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.22.221.66:33253)
0926 07:34:17.270023 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:17.270030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:17.270056 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:34:17.270116 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:17.270122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:17.270324 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:17.270458 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:17.271710 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.272681 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.273878 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.274387 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.274492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.278422 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:17.278433 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:17.278456 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:17.278495 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:17.281390 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.281989 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.281995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.281998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.282121 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.282405 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.282408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.282410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.282716 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:17.282936 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:17.283125 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:34:17.283451 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
I20260926 07:34:17.285027 32003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:17.366133 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 31880
I20260926 07:34:17.383560 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32013
Sep 26 07:34:17 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:34:17.400516 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:17.384524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.22.221.67:46077)
0926 07:34:17.385164 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:17.385170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:17.385199 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:34:17.385261 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:17.385275 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:17.385483 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:17.385667 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:17.387026 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.388250 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.389648 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.390184 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.390352 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.394723 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:17.394733 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:17.394746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:17.394778 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:17.397540 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.398350 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.398373 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.398375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.398502 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.398895 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.398900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.398904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.399219 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:17.399609 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:17.399895 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 07:34:17.400146 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":341,"threads_started":1}
I20260926 07:34:17.401768 32136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:17.486821 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 32013
I20260926 07:34:17.502734 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 31653
I20260926 07:34:17.504009 31715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:17.606259 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 31653
I20260926 07:34:17.626430 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:33337
--webserver_interface=127.22.221.126
--webserver_port=44065
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:33337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:17.942261 32158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:17.942589 32158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:17.942749 32158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:17.949760 32158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:17.949901 32158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:17.950011 32158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:17.950116 32158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:17.950205 32158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:17.959784 32158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:33337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:33337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=44065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:17.961063 32158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:17.962349 32158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:17.973805 32166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:17.973754 32164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:17.973891 32163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:17.974582 32158 server_base.cc:1034] running on GCE node
I20260926 07:34:17.975312 32158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:17.976660 32158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:17.977855 32158 hybrid_clock.cc:648] HybridClock initialized: now 1790408057977814 us; error 28 us; skew 500 ppm
I20260926 07:34:17.978194 32158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:17 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:17.981254 32158 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:34:17.982571 32158 webserver.cc:460] Webserver started at http://127.22.221.126:44065/ using document root <none> and password file <none>
I20260926 07:34:17.983163 32158 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:17.983330 32158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:17.988363 32158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:34:17.991122 32173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:17.992516 32158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:17.992697 32158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "105370ee1a324c5bb8da68e2758b36a3"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "cdf37fbae33f34b8091772a24a94b341"
server_key_iv: "adcf2d7abe9c2e3f1460e08a5a920819"
server_key_version: "encryptionkey@0"
I20260926 07:34:17.993670 32158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:18.008064 32158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:18.012283 32158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:18.012562 32158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:18.030212 32158 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:33337
I20260926 07:34:18.030233 32224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:33337 every 8 connection(s)
I20260926 07:34:18.031674 32158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:18.036778 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 32158
I20260926 07:34:18.037761 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:42359
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:33337
--webserver_port=34795
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:18.039036 32225 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:34:18.044070 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Bootstrap starting.
I20260926 07:34:18.062964 32225 log.cc:824] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:18.071429 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:34:18.071945 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Bootstrap complete.
I20260926 07:34:18.080775 32225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:18.081542 32225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105370ee1a324c5bb8da68e2758b36a3, State: Initialized, Role: FOLLOWER
I20260926 07:34:18.082252 32225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:18.082495 32225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:18.082603 32225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:18.082788 32225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:34:18.086468 32225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:18.086946 32225 leader_election.cc:302] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105370ee1a324c5bb8da68e2758b36a3; no voters:
I20260926 07:34:18.087620 32225 leader_election.cc:288] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:34:18.087755 32229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:34:18.089015 32229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [term 2 LEADER]: Becoming Leader. State: Replica: 105370ee1a324c5bb8da68e2758b36a3, State: Running, Role: LEADER
I20260926 07:34:18.089581 32229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } }
I20260926 07:34:18.089836 32225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:18.092821 32231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105370ee1a324c5bb8da68e2758b36a3. Latest consensus state: current_term: 2 leader_uuid: "105370ee1a324c5bb8da68e2758b36a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } } }
I20260926 07:34:18.093169 32231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:18.092857 32230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "105370ee1a324c5bb8da68e2758b36a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105370ee1a324c5bb8da68e2758b36a3" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33337 } } }
I20260926 07:34:18.093299 32230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:18.094555 32236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:18.107123 32236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:18.108857 32236 catalog_manager.cc:1259] Loaded cluster ID: d9996d6b29404fa6b982ff92eaaa7d0c
I20260926 07:34:18.108974 32236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:18.114075 32236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:18.115968 32236 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 105370ee1a324c5bb8da68e2758b36a3: Loaded TSK: 0
I20260926 07:34:18.117858 32236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:34:18.387493 32227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:18.387781 32227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:18.387944 32227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:18.395352 32227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:18.395484 32227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:18.395646 32227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:18.405501 32227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.65.keytab
--principal=oryx/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:42359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=34795
--tserver_master_addrs=127.22.221.126:33337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:18.406718 32227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:18.408047 32227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:18.423920 32251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:18.423846 32252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:18.424237 32227 server_base.cc:1034] running on GCE node
W20260926 07:34:18.423871 32254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:18.425074 32227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:18.426002 32227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:18.427270 32227 hybrid_clock.cc:648] HybridClock initialized: now 1790408058427177 us; error 47 us; skew 500 ppm
I20260926 07:34:18.427588 32227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408058, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:18.430716 32227 init.cc:377] Logged in from keytab as oryx/127.22.221.65@KRBTEST.COM (short username oryx)
I20260926 07:34:18.432260 32227 webserver.cc:460] Webserver started at http://127.22.221.65:34795/ using document root <none> and password file <none>
I20260926 07:34:18.432996 32227 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:18.433136 32227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:18.438422 32227 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:34:18.441251 32261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:18.442689 32227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:18.442821 32227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "eba67684f5b347b0849cbb292b555a82"
format_stamp: "Formatted at 2026-09-26 07:34:15 on dist-test-slave-qlq8"
server_key: "a4d56d107ce9953885e593f2a6940083"
server_key_iv: "78fcb6636e599d7823655ede9a022063"
server_key_version: "encryptionkey@0"
I20260926 07:34:18.443805 32227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:18.460268 32227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:18.465226 32227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:18.465520 32227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:18.466799 32227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:18.468647 32227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:18.468761 32227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:18.468861 32227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:18.468945 32227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:18.503799 32227 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:42359
I20260926 07:34:18.503886 32373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:42359 every 8 connection(s)
I20260926 07:34:18.505246 32227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:18.511968 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 32227
I20260926 07:34:18.512755 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:33253
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:33337
--webserver_port=36443
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:34:18.525002 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:18.508075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:53733 (local address 127.22.221.126:33337)
0926 07:34:18.508723 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:18.508734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:18.509046 (+ 312us) server_negotiation.cc:407] Connection header received
0926 07:34:18.511133 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:18.511145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:18.511562 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:18.511858 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:18.513872 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.515047 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.517503 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.518152 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.518290 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.524153 (+ 5863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:53733: BlockingRecv error: recv got EOF from 127.22.221.65:53733 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":275,"threads_started":1}
I20260926 07:34:18.525085 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:18.508097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:53733)
0926 07:34:18.508883 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:18.508933 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:18.510659 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:18.512119 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:18.512133 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:18.512883 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:18.513672 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.513698 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.515221 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.515257 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.517357 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.517375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.518381 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.518390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.518723 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.519521 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:18.519543 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:18.523871 (+ 4328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":349,"threads_started":1}
W20260926 07:34:18.525758 32374 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:33337 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.221.126:33337: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:18.534333 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:18.526357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:46015)
0926 07:34:18.526553 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:18.526572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:18.526687 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:18.527054 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:18.527057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:18.527392 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:18.527529 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.527536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.528967 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.528970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.529874 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.529884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.530856 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.530860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.530973 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.531563 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:18.531579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:18.534155 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 07:34:18.534693 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:18.526553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:46015 (local address 127.22.221.126:33337)
0926 07:34:18.526757 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:18.526765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:18.526785 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:18.526837 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:18.526844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:18.526919 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:18.527027 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:18.527664 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.528837 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.530014 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.530719 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.530863 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.534425 (+ 3562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:46015: BlockingRecv error: recv got EOF from 127.22.221.65:46015 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:18.542757 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:18.535043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:33237)
0926 07:34:18.535206 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:18.535221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:18.535339 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:18.535641 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:18.535644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:18.535894 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:18.536023 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.536030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.537523 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.537526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.538388 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.538397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.539333 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.539337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.539482 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.540051 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:18.540067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:18.542571 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:34:18.543032 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:18.535124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:33237 (local address 127.22.221.126:33337)
0926 07:34:18.535324 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:18.535332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:18.535355 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:34:18.535413 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:18.535420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:18.535513 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:18.535627 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:18.536162 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.537380 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.538534 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.539162 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.539303 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.542787 (+ 3484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:33237: BlockingRecv error: recv got EOF from 127.22.221.65:33237 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 07:34:18.543126 32374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:34:18.832469 32378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:18.832746 32378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:18.832845 32378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:18.839903 32378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:18.839996 32378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:18.840101 32378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:18.849409 32378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.66.keytab
--principal=oryx/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:33253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=36443
--tserver_master_addrs=127.22.221.126:33337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:18.850528 32378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:18.851778 32378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:18.864701 32383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:18.864696 32384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:18.864727 32386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:18.865633 32378 server_base.cc:1034] running on GCE node
I20260926 07:34:18.866170 32378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:18.867065 32378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:18.868325 32378 hybrid_clock.cc:648] HybridClock initialized: now 1790408058868294 us; error 44 us; skew 500 ppm
I20260926 07:34:18.868646 32378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408058, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:18.871618 32378 init.cc:377] Logged in from keytab as oryx/127.22.221.66@KRBTEST.COM (short username oryx)
I20260926 07:34:18.873039 32378 webserver.cc:460] Webserver started at http://127.22.221.66:36443/ using document root <none> and password file <none>
I20260926 07:34:18.873679 32378 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:18.873832 32378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:18.878892 32378 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:18.881419 32393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:18.882591 32378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:34:18.882757 32378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "5efdf74edffe48848be0a6f9a10396f1"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "f7bc2d3941778dfdfeee036d12c4641b"
server_key_iv: "3ccbac1037fc19c5ee6a177e8f0d48cb"
server_key_version: "encryptionkey@0"
I20260926 07:34:18.883726 32378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:18.899706 32378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:18.903949 32378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:18.904175 32378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:18.905001 32378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:18.906255 32378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:18.906340 32378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:18.906411 32378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:18.906471 32378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:18.936873 32378 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:33253
I20260926 07:34:18.936959 32505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:33253 every 8 connection(s)
I20260926 07:34:18.938294 32378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:18.946789 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 32378
I20260926 07:34:18.947582 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:46077
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:33337
--webserver_port=36101
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:34:18.954010 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:18.940837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:33809 (local address 127.22.221.126:33337)
0926 07:34:18.941023 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:18.941027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:18.942066 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 07:34:18.943954 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:18.943959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:18.944049 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:18.944159 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:18.945577 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.946289 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.948235 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.948720 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.948853 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.953823 (+ 4970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:33809: BlockingRecv error: recv got EOF from 127.22.221.66:33809 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:18.954952 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:18.941249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:33809)
0926 07:34:18.941907 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:18.941945 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:18.943681 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:18.944383 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:18.944391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:18.944870 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:18.945404 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.945421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.946456 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.946460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.948095 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.948107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.948948 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.948957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.949211 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.949890 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:18.949913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:18.953666 (+ 3753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":266,"threads_started":1}
W20260926 07:34:18.955531 32506 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:33337 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.221.126:33337: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:18.964097 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:18.956117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:54119)
0926 07:34:18.956274 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:18.956294 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:18.956394 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:18.956763 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:18.956766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:18.957052 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:18.957171 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.957178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.958599 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.958602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.959540 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.959551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.960566 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.960570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.960680 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.961234 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:18.961250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:18.963917 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 07:34:18.964502 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:18.956269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:54119 (local address 127.22.221.126:33337)
0926 07:34:18.956466 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:18.956474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:18.956494 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:18.956549 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:18.956557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:18.956636 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:18.956737 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:18.957311 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.958475 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.959676 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.960420 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.960562 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.964212 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:54119: BlockingRecv error: recv got EOF from 127.22.221.66:54119 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:18 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:18.973228 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:18.964911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:57113)
0926 07:34:18.965075 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:18.965092 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:18.965170 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:18.965519 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:18.965521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:18.965772 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:18.965897 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.965904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.967456 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.967463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.968614 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:18.968626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.969758 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.969765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:18.969925 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.970472 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:18.970489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:18.973044 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:34:18.973475 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:18.965020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:57113 (local address 127.22.221.126:33337)
0926 07:34:18.965218 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:18.965226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:18.965246 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:18.965301 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:18.965308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:18.965389 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:18.965507 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:18.966050 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.967305 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.968760 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:18.969600 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:18.969751 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:18.973274 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:57113: BlockingRecv error: recv got EOF from 127.22.221.66:57113 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:34:18.973630 32506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:34:19.271716 32509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:19.271971 32509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:19.272136 32509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:19.279132 32509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:19.279270 32509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:19.279441 32509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:19.288729 32509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/krb5kdc/oryx_127.22.221.67.keytab
--principal=oryx/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:46077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=36101
--tserver_master_addrs=127.22.221.126:33337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:19.289947 32509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:19.291221 32509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:19.304214 32514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:19.304279 32515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:19.304303 32517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:19.305147 32509 server_base.cc:1034] running on GCE node
I20260926 07:34:19.305697 32509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:19.306609 32509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:19.307826 32509 hybrid_clock.cc:648] HybridClock initialized: now 1790408059307782 us; error 45 us; skew 500 ppm
I20260926 07:34:19.308148 32509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408059, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:19.311287 32509 init.cc:377] Logged in from keytab as oryx/127.22.221.67@KRBTEST.COM (short username oryx)
I20260926 07:34:19.312697 32509 webserver.cc:460] Webserver started at http://127.22.221.67:36101/ using document root <none> and password file <none>
I20260926 07:34:19.313309 32509 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:19.313417 32509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:19.318380 32509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:34:19.320952 32524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:19.322217 32509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:34:19.322346 32509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "233a7f4d4338421e99c766fd6b376e69"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-qlq8"
server_key: "14f24ec9f41f969c80f1efc5f705672f"
server_key_iv: "4280301fb211b6095ba8076101558977"
server_key_version: "encryptionkey@0"
I20260926 07:34:19.323287 32509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:19.337000 32509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:19.340579 32509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:19.340786 32509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:19.341611 32509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:19.342743 32509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:19.342833 32509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:19.342907 32509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:19.342962 32509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:19.376004 32509 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:46077
I20260926 07:34:19.376088 32636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:46077 every 8 connection(s)
I20260926 07:34:19.377381 32509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:19.380198 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 32509
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:34:19.396334 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:19.379791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:38703 (local address 127.22.221.126:33337)
0926 07:34:19.379963 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.379968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.381442 (+ 1474us) server_negotiation.cc:407] Connection header received
0926 07:34:19.383712 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.383717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.383809 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.383923 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.385720 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.386898 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.388973 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.389680 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.389882 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.396047 (+ 6165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:38703: BlockingRecv error: recv got EOF from 127.22.221.67:38703 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:19.396955 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.380372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:38703)
0926 07:34:19.381264 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:19.381318 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:19.383460 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:19.384140 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:19.384148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:19.384813 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:19.385544 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.385566 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.387040 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.387044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.388821 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.388838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.390030 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.390038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.390249 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.390865 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:19.390887 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:19.395793 (+ 4906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":374,"threads_started":1}
W20260926 07:34:19.397629 32637 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:33337 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.221.126:33337: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
I20260926 07:34:19.399066 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.381271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.22.221.126:33337)
0926 07:34:19.382166 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.382175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.382742 (+ 567us) server_negotiation.cc:407] Connection header received
0926 07:34:19.382874 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.382881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.382970 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.383089 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.384598 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.385692 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.387535 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.388179 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.388303 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.391311 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:19.391323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:19.391344 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:19.391390 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:19.395309 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:19.396165 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:19.396174 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:19.396182 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:19.396408 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:19.396887 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:19.396891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:19.396894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:19.397506 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:19.397823 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:19.398296 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 07:34:19.398642 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":598,"threads_started":1}
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:19.406313 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.398205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:46397)
0926 07:34:19.398429 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:19.398447 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:19.398552 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:19.398882 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:19.398886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:19.399218 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:19.399404 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.399415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.400769 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.400773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.401934 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.401952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.402889 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.402893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.403008 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.403506 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:19.403522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:19.406141 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 07:34:19.406585 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:19.398268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:46397 (local address 127.22.221.126:33337)
0926 07:34:19.398415 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.398421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.398493 (+ 72us) server_negotiation.cc:407] Connection header received
0926 07:34:19.398657 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.398662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.398755 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.398859 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.399548 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.400571 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.402072 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.402720 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.402855 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.406324 (+ 3469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:46397: BlockingRecv error: recv got EOF from 127.22.221.67:46397 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:19.413970 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.406903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:36059)
0926 07:34:19.407058 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:19.407098 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:19.407188 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:19.407503 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:19.407507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:19.407796 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:19.407921 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.407929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.408868 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.408874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.409751 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.409761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.410402 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.410408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.410600 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.411053 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:19.411068 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:19.413663 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:34:19.414045 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:19.406997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:36059 (local address 127.22.221.126:33337)
0926 07:34:19.407203 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.407208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.407281 (+ 73us) server_negotiation.cc:407] Connection header received
0926 07:34:19.407326 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.407330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.407389 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.407482 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.408050 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.408737 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.409888 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.410270 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.410391 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.413829 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:36059: BlockingRecv error: recv got EOF from 127.22.221.67:36059 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260926 07:34:19.414253 32637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:19.552208 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.543804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:40079)
0926 07:34:19.544310 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:19.544325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:19.544481 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:19.544816 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:19.544821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:19.545302 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:19.545490 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.545499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.546529 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.546536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.547595 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.547607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.548344 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.548352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.548512 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.549131 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:19.549148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:19.551865 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
W20260926 07:34:19.552287 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:19.543862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:40079 (local address 127.22.221.126:33337)
0926 07:34:19.544046 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.544054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.544408 (+ 354us) server_negotiation.cc:407] Connection header received
0926 07:34:19.544612 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.544623 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.544683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.544780 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.545633 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.546358 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.547738 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.548160 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.548287 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.552071 (+ 3784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:40079: BlockingRecv error: recv got EOF from 127.22.221.65:40079 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:19 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:19.982409 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.974197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:36935)
0926 07:34:19.974721 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:19.974736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:19.974852 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:19.975141 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:19.975144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:19.975628 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:19.975818 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.975827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.976737 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.976740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.977673 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:19.977684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.978385 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.978389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:19.978576 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.979321 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:19.979339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:19.982189 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
W20260926 07:34:19.982620 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:19.974341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:36935 (local address 127.22.221.126:33337)
0926 07:34:19.974548 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.974552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.974811 (+ 259us) server_negotiation.cc:407] Connection header received
0926 07:34:19.974957 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.974960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.975018 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.975099 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.975942 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.976615 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.977791 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.978232 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.978357 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.982383 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:36935: BlockingRecv error: recv got EOF from 127.22.221.66:36935 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 07:34:20 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:20 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:20.423444 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:20.414911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:38213)
0926 07:34:20.415620 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:20.415640 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:20.415752 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:20.416017 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:20.416020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:20.416453 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:20.416638 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:20.416647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.417627 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.417631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:20.418609 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:20.418620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.419408 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.419417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:20.419580 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:20.420112 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:20.420128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:20.423083 (+ 2955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":499,"threads_started":1}
W20260926 07:34:20.423537 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:20.414998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:38213 (local address 127.22.221.126:33337)
0926 07:34:20.415272 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:20.415279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:20.415721 (+ 442us) server_negotiation.cc:407] Connection header received
0926 07:34:20.415848 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:20.415852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:20.415915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:20.415995 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:20.416757 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.417506 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.418753 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.419179 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.419359 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:20.423299 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:38213: BlockingRecv error: recv got EOF from 127.22.221.67:38213 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 07:34:20 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:20 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:20.561465 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:20.553370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:56195)
0926 07:34:20.553917 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:20.553945 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:20.554072 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:20.554464 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:20.554468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:20.554995 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:20.555163 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:20.555172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.556196 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.556199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:20.557083 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:20.557093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.557708 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.557712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:20.557821 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:20.558356 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:20.558373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:20.561112 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
W20260926 07:34:20.561688 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:20.553535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:56195 (local address 127.22.221.126:33337)
0926 07:34:20.553743 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:20.553748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:20.554049 (+ 301us) server_negotiation.cc:407] Connection header received
0926 07:34:20.554188 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:20.554193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:20.554295 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:20.554412 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:20.555318 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.556082 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.557201 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.557581 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.557677 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:20.561305 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:56195: BlockingRecv error: recv got EOF from 127.22.221.65:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:34:20 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:20 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:20.992090 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:20.983420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:40139)
0926 07:34:20.983924 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:20.983941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:20.984052 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:20.984435 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:20.984439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:20.984846 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:20.985046 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:20.985055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.986326 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.986333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:20.987438 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:20.987451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.988122 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.988126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:20.988253 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:20.988845 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:20.988861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:20.991721 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
W20260926 07:34:20.992116 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:20.983492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:40139 (local address 127.22.221.126:33337)
0926 07:34:20.983725 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:20.983731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:20.984021 (+ 290us) server_negotiation.cc:407] Connection header received
0926 07:34:20.984150 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:20.984153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:20.984214 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:20.984296 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:20.985203 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.986209 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.987571 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:20.987978 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:20.988123 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:20.991905 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:40139: BlockingRecv error: recv got EOF from 127.22.221.66:40139 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 07:34:21 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:21 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:21.434082 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:21.424501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:43895)
0926 07:34:21.425258 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:21.425282 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:21.425446 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:21.425813 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:21.425818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:21.426245 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:21.426406 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:21.426423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.427580 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.427585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:21.428586 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:21.428597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.429331 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.429337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:21.429513 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:21.430206 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:21.430228 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:21.433725 (+ 3497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
W20260926 07:34:21.434180 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:21.424703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:43895 (local address 127.22.221.126:33337)
0926 07:34:21.424965 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:21.424971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:21.425369 (+ 398us) server_negotiation.cc:407] Connection header received
0926 07:34:21.425572 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:21.425577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:21.425676 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:21.425779 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:21.426556 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.427436 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.428737 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.429180 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.429320 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:21.433927 (+ 4607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:43895: BlockingRecv error: recv got EOF from 127.22.221.67:43895 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 26 07:34:21 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:21 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:21.570365 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:21.562548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:47131)
0926 07:34:21.562985 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:21.563001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:21.563113 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:21.563433 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:21.563436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:21.563759 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:21.563924 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:21.563944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.564911 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.564914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:21.565819 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:21.565830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.566650 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.566655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:21.566842 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:21.567519 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:21.567536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:21.570120 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":239,"threads_started":1}
W20260926 07:34:21.570539 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:21.562605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:47131 (local address 127.22.221.126:33337)
0926 07:34:21.562841 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:21.562848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:21.563101 (+ 253us) server_negotiation.cc:407] Connection header received
0926 07:34:21.563212 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:21.563215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:21.563295 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:21.563381 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:21.564059 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.564786 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.565986 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.566499 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.566628 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:21.570295 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:47131: BlockingRecv error: recv got EOF from 127.22.221.65:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 07:34:21 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:21 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:22.001441 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:21.992956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:58367)
0926 07:34:21.993436 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:21.993451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:21.993604 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:21.993934 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:21.993938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:21.994296 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:21.994468 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:21.994477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.995576 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.995587 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:21.996472 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:21.996483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.997232 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.997240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:21.997432 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:21.998086 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:21.998104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:22.001184 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":261,"threads_started":1}
W20260926 07:34:22.001598 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:21.993025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:58367 (local address 127.22.221.126:33337)
0926 07:34:21.993246 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:21.993255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:21.993532 (+ 277us) server_negotiation.cc:407] Connection header received
0926 07:34:21.993722 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:21.993728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:21.993798 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:21.993898 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:21.994609 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.995442 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.996611 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:21.997088 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:21.997261 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.001361 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:58367: BlockingRecv error: recv got EOF from 127.22.221.66:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 07:34:22 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:22 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:22.448521 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:22.435053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:51855)
0926 07:34:22.436902 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:22.436930 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:22.437085 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:22.437457 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:22.437462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:22.437834 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:22.438000 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:22.438009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.439008 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.439015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:22.440010 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:22.440022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.440722 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.440728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:22.440918 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.444987 (+ 4069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:22.445004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:22.448158 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":1621,"threads_started":1}
W20260926 07:34:22.448597 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:22.435145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:51855 (local address 127.22.221.126:33337)
0926 07:34:22.435362 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:22.435368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:22.437018 (+ 1650us) server_negotiation.cc:407] Connection header received
0926 07:34:22.437219 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:22.437225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:22.437323 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:22.437432 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:22.438142 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.438880 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.440157 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.440578 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.440710 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.448356 (+ 7646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:51855: BlockingRecv error: recv got EOF from 127.22.221.67:51855 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 07:34:22 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:22 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:22.580034 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:22.571521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:49351)
0926 07:34:22.572016 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:22.572031 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:22.572194 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:22.572542 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:22.572547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:22.572958 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:22.573105 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:22.573114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.574120 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.574127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:22.575106 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:22.575117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.575841 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.575849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:22.576036 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.576605 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:22.576621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:22.579810 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
W20260926 07:34:22.580219 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:22.571590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:49351 (local address 127.22.221.126:33337)
0926 07:34:22.571843 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:22.571851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:22.572121 (+ 270us) server_negotiation.cc:407] Connection header received
0926 07:34:22.572320 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:22.572331 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:22.572390 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:22.572505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:22.573255 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.573989 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.575272 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.575693 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.575829 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.579999 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:49351: BlockingRecv error: recv got EOF from 127.22.221.65:49351 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 26 07:34:23 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:23 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:23.010921 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:23.002539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:47791)
0926 07:34:23.003109 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:23.003132 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:23.003308 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:23.003713 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:23.003717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:23.004163 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:23.004319 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:23.004328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.005298 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.005302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:23.006253 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:23.006265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.006917 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.006921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:23.007035 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.007737 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:23.007755 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:23.010693 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":305,"threads_started":1}
W20260926 07:34:23.011101 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:23.002579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:47791 (local address 127.22.221.126:33337)
0926 07:34:23.002782 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:23.002788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:23.003270 (+ 482us) server_negotiation.cc:407] Connection header received
0926 07:34:23.003414 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:23.003441 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:23.003543 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:23.003662 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:23.004458 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.005186 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.006378 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.006789 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.006879 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.010880 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:47791: BlockingRecv error: recv got EOF from 127.22.221.66:47791 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:34:23 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:23 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:23.457832 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:23.449486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:42737)
0926 07:34:23.449978 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:23.450001 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:23.450126 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:23.450391 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:23.450394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:23.450755 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:23.450898 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:23.450907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.452088 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.452096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:23.453389 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:23.453402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.454090 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.454096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:23.454273 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.454828 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:23.454845 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:23.457505 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":278,"threads_started":1}
W20260926 07:34:23.457959 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:23.449605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:42737 (local address 127.22.221.126:33337)
0926 07:34:23.449771 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:23.449775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:23.450085 (+ 310us) server_negotiation.cc:407] Connection header received
0926 07:34:23.450208 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:23.450212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:23.450270 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:23.450350 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:23.451025 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.451955 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.453536 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.453946 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.454075 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.457717 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:42737: BlockingRecv error: recv got EOF from 127.22.221.67:42737 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:34:23 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:23 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:23.589406 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:23.581155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:51893)
0926 07:34:23.581795 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:23.581817 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:23.581978 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:23.582363 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:23.582371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:23.582798 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:23.582972 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:23.582980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.584016 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.584020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:23.584937 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:23.584948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.585607 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.585611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:23.585742 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.586290 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:23.586307 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:23.589167 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":315,"threads_started":1}
W20260926 07:34:23.589589 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:23.581221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:51893 (local address 127.22.221.126:33337)
0926 07:34:23.581503 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:23.581512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:23.581907 (+ 395us) server_negotiation.cc:407] Connection header received
0926 07:34:23.582104 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:23.582109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:23.582202 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:23.582303 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:23.583121 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.583874 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.585059 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.585488 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.585579 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.589346 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:51893: BlockingRecv error: recv got EOF from 127.22.221.65:51893 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 07:34:24 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:24 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:24.020363 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:24.011983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:46581)
0926 07:34:24.012513 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:24.012529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:24.012633 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:24.012925 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:24.012928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:24.013360 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:24.013536 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:24.013546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.014495 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.014498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:24.015437 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:24.015448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.016387 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.016393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:24.016589 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:24.017243 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:24.017262 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:24.020017 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":274,"threads_started":1}
W20260926 07:34:24.020467 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:24.012161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:46581 (local address 127.22.221.126:33337)
0926 07:34:24.012325 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:24.012329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:24.012599 (+ 270us) server_negotiation.cc:407] Connection header received
0926 07:34:24.012733 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:24.012737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:24.012799 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:24.012881 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:24.013661 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.014383 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.015590 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.016256 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.016378 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:24.020218 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:46581: BlockingRecv error: recv got EOF from 127.22.221.66:46581 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:34:24 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:24 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:24.468317 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:24.458904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:44751)
0926 07:34:24.459439 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:24.459459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:24.459571 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:24.459838 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:24.459841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:24.460249 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:24.460408 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:24.460417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.461363 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.461366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:24.462299 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:24.462318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.463038 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.463044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:24.463223 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:24.463973 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:24.464001 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:24.468002 (+ 4001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":251,"threads_started":1}
W20260926 07:34:24.468499 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:24.459010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:44751 (local address 127.22.221.126:33337)
0926 07:34:24.459281 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:24.459287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:24.459542 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:34:24.459655 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:24.459658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:24.459717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:24.459794 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:24.460539 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.461240 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.462458 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.462883 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.463028 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:24.468225 (+ 5197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:44751: BlockingRecv error: recv got EOF from 127.22.221.67:44751 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 07:34:24 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:24 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:24.598524 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:24.590362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:53081)
0926 07:34:24.590937 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:24.590958 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:24.591117 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:24.591501 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:24.591505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:24.591858 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:24.592015 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:24.592023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.593030 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.593033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:24.593942 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:24.593952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.594671 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.594677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:24.594857 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:24.595557 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:24.595574 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:24.598301 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":318,"threads_started":1}
W20260926 07:34:24.598728 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:24.590495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:53081 (local address 127.22.221.126:33337)
0926 07:34:24.590735 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:24.590744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:24.591052 (+ 308us) server_negotiation.cc:407] Connection header received
0926 07:34:24.591276 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:24.591280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:24.591342 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:24.591460 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:24.592145 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.592907 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.594078 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:24.594510 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:24.594650 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:24.598469 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:53081: BlockingRecv error: recv got EOF from 127.22.221.65:53081 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 07:34:25 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:25 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:25.029477 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:25.021419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:36623)
0926 07:34:25.021968 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:25.021984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:25.022095 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:25.022382 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:25.022385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:25.022743 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:25.022904 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:25.022915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.023911 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.023916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:25.024801 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:25.024812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.025474 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.025478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:25.025601 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:25.026222 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:25.026239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:25.029127 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":312,"threads_started":1}
W20260926 07:34:25.029639 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:25.021462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:36623 (local address 127.22.221.126:33337)
0926 07:34:25.021639 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:25.021644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:25.022064 (+ 420us) server_negotiation.cc:407] Connection header received
0926 07:34:25.022191 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:25.022195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:25.022255 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:25.022333 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:25.023041 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.023785 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.024924 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.025328 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.025427 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:25.029361 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:36623: BlockingRecv error: recv got EOF from 127.22.221.66:36623 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 07:34:25 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:25 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:25.477443 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:25.469316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:43467)
0926 07:34:25.469798 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:25.469819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:25.469934 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:25.470215 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:25.470218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:25.470581 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:25.470735 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:25.470743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.471836 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.471840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:25.472757 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:25.472769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.473460 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.473464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:25.473591 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:25.474250 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:25.474268 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:25.477234 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
W20260926 07:34:25.477650 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:25.469404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:43467 (local address 127.22.221.126:33337)
0926 07:34:25.469636 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:25.469642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:25.469915 (+ 273us) server_negotiation.cc:407] Connection header received
0926 07:34:25.470030 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:25.470033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:25.470095 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:25.470173 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:25.470853 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.471713 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.472891 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.473319 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.473409 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:25.477421 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:43467: BlockingRecv error: recv got EOF from 127.22.221.67:43467 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 07:34:25 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:25 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:25.608196 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:25.599522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:51133)
0926 07:34:25.600097 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:25.600113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:25.600252 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:25.600587 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:25.600590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:25.601002 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:25.601162 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:25.601170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.602240 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.602244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:25.603153 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:25.603164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.603978 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.603982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:25.604093 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:25.604624 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:25.604640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:25.607843 (+ 3203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":248,"threads_started":1}
W20260926 07:34:25.608285 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:25.599526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:51133 (local address 127.22.221.126:33337)
0926 07:34:25.599768 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:25.599775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:25.600194 (+ 419us) server_negotiation.cc:407] Connection header received
0926 07:34:25.600399 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:25.600403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:25.600468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:25.600549 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:25.601282 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.602116 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.603310 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:25.603838 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:25.603941 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:25.608062 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:51133: BlockingRecv error: recv got EOF from 127.22.221.65:51133 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 07:34:26 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:26 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:26.039019 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:26.030525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:54681)
0926 07:34:26.031078 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:26.031093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:26.031287 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:26.031660 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:26.031665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:26.032125 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:26.032290 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:26.032300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.033376 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.033382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:26.034327 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:26.034338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.035084 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.035090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:26.035313 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.035946 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:26.035963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:26.038649 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":259,"threads_started":1}
W20260926 07:34:26.039057 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:26.030617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:54681 (local address 127.22.221.126:33337)
0926 07:34:26.030852 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:26.030861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:26.031176 (+ 315us) server_negotiation.cc:407] Connection header received
0926 07:34:26.031426 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:26.031431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:26.031521 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:26.031622 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:26.032447 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.033237 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.034481 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.034929 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.035054 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.038833 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:54681: BlockingRecv error: recv got EOF from 127.22.221.66:54681 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 26 07:34:26 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:26 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:26.486013 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:26.478344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:52369)
0926 07:34:26.478867 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:26.478903 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:26.479017 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:26.479329 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:26.479332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:26.479740 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:26.479917 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:26.479927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.480871 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.480875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:26.481774 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:26.481784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.482441 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.482445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:26.482556 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.483153 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:26.483170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:26.485756 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":232,"threads_started":1}
W20260926 07:34:26.486153 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:26.478388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:52369 (local address 127.22.221.126:33337)
0926 07:34:26.478591 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:26.478601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:26.478979 (+ 378us) server_negotiation.cc:407] Connection header received
0926 07:34:26.479098 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:26.479101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:26.479162 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:26.479284 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:26.480038 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.480756 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.481898 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.482315 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.482416 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.485939 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:52369: BlockingRecv error: recv got EOF from 127.22.221.67:52369 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 07:34:26 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:26 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:26.617816 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:26.609154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:43375)
0926 07:34:26.609713 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:26.609729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:26.609879 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:26.610232 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:26.610237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:26.610674 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:26.610828 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:26.610836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.611940 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.611946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:26.612879 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:26.612890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.613649 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.613656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:26.613832 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.614552 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:26.614568 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:26.617588 (+ 3020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":304,"threads_started":1}
W20260926 07:34:26.618008 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:26.609341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:43375 (local address 127.22.221.126:33337)
0926 07:34:26.609579 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:26.609586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:26.609814 (+ 228us) server_negotiation.cc:407] Connection header received
0926 07:34:26.610023 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:26.610028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:26.610100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:26.610201 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:26.610974 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.611783 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.613027 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.613471 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.613614 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.617759 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:43375: BlockingRecv error: recv got EOF from 127.22.221.65:43375 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 07:34:27 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:27 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:27.059106 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:27.040130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:58053)
0926 07:34:27.044216 (+ 4086us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:27.044236 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:27.044382 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:27.044780 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:27.044785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:27.045306 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:27.045527 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:27.045539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.051338 (+ 5799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.051344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:27.052996 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:27.053016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.054150 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.054159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:27.054404 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:27.055192 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:27.055216 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:27.058689 (+ 3473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3936,"thread_start_us":3809,"threads_started":1}
W20260926 07:34:27.059274 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:27.040201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:58053 (local address 127.22.221.126:33337)
0926 07:34:27.040418 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:27.040427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:27.044316 (+ 3889us) server_negotiation.cc:407] Connection header received
0926 07:34:27.044491 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:27.044496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:27.044599 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:27.044750 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:27.045686 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.047054 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.053260 (+ 6206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.053947 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.054073 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:27.058969 (+ 4896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:58053: BlockingRecv error: recv got EOF from 127.22.221.66:58053 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:34:27 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:27 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:27.500931 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:27.487137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:43791)
0926 07:34:27.487708 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:27.487727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:27.487904 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:27.488246 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:27.488251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:27.488651 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:27.488800 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:27.488808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.489857 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.489862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:27.491356 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:27.491374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.492168 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.492172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:27.492280 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:27.497999 (+ 5719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:27.498016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:27.500686 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":292,"threads_started":1}
W20260926 07:34:27.501127 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:27.487249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:43791 (local address 127.22.221.126:33337)
0926 07:34:27.487502 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:27.487509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:27.487811 (+ 302us) server_negotiation.cc:407] Connection header received
0926 07:34:27.488035 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:27.488040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:27.488111 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:27.488223 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:27.488952 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.489687 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.491541 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.492011 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.492111 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:27.500888 (+ 8777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:43791: BlockingRecv error: recv got EOF from 127.22.221.67:43791 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 07:34:27 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:27 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:27.627344 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:27.618852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:46355)
0926 07:34:27.619418 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:27.619435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:27.619549 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:27.619830 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:27.619833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:27.620215 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:27.620375 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:27.620384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.621730 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.621756 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:27.622749 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:27.622761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.623426 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.623430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:27.623546 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:27.624163 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:27.624180 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:27.627047 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":281,"threads_started":1}
W20260926 07:34:27.627521 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:27.619026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:46355 (local address 127.22.221.126:33337)
0926 07:34:27.619300 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:27.619307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:27.619514 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:34:27.619643 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:27.619647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:27.619709 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:27.619789 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:27.620523 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.621595 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.622873 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:27.623290 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:27.623434 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:27.627220 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:46355: BlockingRecv error: recv got EOF from 127.22.221.65:46355 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 07:34:28 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:28 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:28.068110 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:28.060273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:56739)
0926 07:34:28.060834 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:28.060850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:28.060972 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:28.061253 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:28.061256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:28.061634 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:28.061794 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:28.061803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.062780 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.062784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:28.063737 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:28.063749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.064414 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.064418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:28.064535 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.065103 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:28.065121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:28.067862 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":323,"threads_started":1}
W20260926 07:34:28.068248 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:28.060370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:56739 (local address 127.22.221.126:33337)
0926 07:34:28.060557 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:28.060561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:28.060938 (+ 377us) server_negotiation.cc:407] Connection header received
0926 07:34:28.061068 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:28.061071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:28.061131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:28.061210 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:28.061934 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.062672 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.063872 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.064287 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.064379 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.068041 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:56739: BlockingRecv error: recv got EOF from 127.22.221.66:56739 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 07:34:28 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:28 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:28.510471 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:28.501853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:60583)
0926 07:34:28.502420 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:28.502442 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:28.502563 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:28.502844 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:28.502847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:28.503252 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:28.503443 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:28.503452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.504571 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.504575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:28.505491 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:28.505502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.506409 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.506418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:28.506565 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.507220 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:28.507286 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:28.510242 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":305,"threads_started":1}
W20260926 07:34:28.510617 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:28.501978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:60583 (local address 127.22.221.126:33337)
0926 07:34:28.502155 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:28.502160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:28.502526 (+ 366us) server_negotiation.cc:407] Connection header received
0926 07:34:28.502663 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:28.502666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:28.502727 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:28.502808 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:28.503566 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.504441 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.505630 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.506244 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.506361 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.510422 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:60583: BlockingRecv error: recv got EOF from 127.22.221.67:60583 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 07:34:28 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:28 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:28.636538 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:28.628263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:42051)
0926 07:34:28.628761 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:28.628776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:28.628942 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:28.629280 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:28.629284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:28.629687 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:28.629828 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:28.629836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.630859 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.630874 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:28.631839 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:28.631851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.632612 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.632617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:28.632799 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.633381 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:28.633397 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:28.636237 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":271,"threads_started":1}
W20260926 07:34:28.636883 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:28.628331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:42051 (local address 127.22.221.126:33337)
0926 07:34:28.628555 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:28.628563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:28.628864 (+ 301us) server_negotiation.cc:407] Connection header received
0926 07:34:28.629086 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:28.629090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:28.629151 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:28.629249 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:28.629984 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.630731 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.631990 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.632448 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.632585 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.636468 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:42051: BlockingRecv error: recv got EOF from 127.22.221.65:42051 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 07:34:29 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:29 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:29.077915 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:29.069135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:54843)
0926 07:34:29.069707 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:29.069723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:29.069832 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:29.070127 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:29.070130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:29.070482 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:29.070627 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:29.070635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.071657 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.071661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:29.073000 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:29.073016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.073980 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.073984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:29.074095 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:29.074773 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:29.074790 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:29.077689 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":384,"threads_started":1}
W20260926 07:34:29.078107 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:29.069211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:54843 (local address 127.22.221.126:33337)
0926 07:34:29.069444 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:29.069453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:29.069806 (+ 353us) server_negotiation.cc:407] Connection header received
0926 07:34:29.069937 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:29.069941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:29.069998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:29.070075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:29.070749 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.071536 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.073167 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.073829 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.073930 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:29.077873 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:54843: BlockingRecv error: recv got EOF from 127.22.221.66:54843 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 07:34:29 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:29 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:29.519774 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:29.511470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:48375)
0926 07:34:29.512001 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:29.512023 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:29.512179 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:29.512510 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:29.512514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:29.512922 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:29.513063 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:29.513071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.514293 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.514298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:29.515221 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:29.515257 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.515897 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.515902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:29.516016 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:29.516571 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:29.516587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:29.519456 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
W20260926 07:34:29.519899 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:29.511644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:48375 (local address 127.22.221.126:33337)
0926 07:34:29.511911 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:29.511918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:29.512105 (+ 187us) server_negotiation.cc:407] Connection header received
0926 07:34:29.512302 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:29.512311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:29.512380 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:29.512483 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:29.513210 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.514133 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.515371 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.515753 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.515842 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:29.519663 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:48375: BlockingRecv error: recv got EOF from 127.22.221.67:48375 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 26 07:34:29 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:29 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:29.645547 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:29.637372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:52159)
0926 07:34:29.637865 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:29.637880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:29.638034 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:29.638385 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:29.638390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:29.638817 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:29.639001 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:29.639010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.640066 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.640073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:29.640968 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:29.640980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.641741 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.641748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:29.641925 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:29.642470 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:29.642486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:29.645206 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
W20260926 07:34:29.645668 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:29.637465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:52159 (local address 127.22.221.126:33337)
0926 07:34:29.637694 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:29.637700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:29.637968 (+ 268us) server_negotiation.cc:407] Connection header received
0926 07:34:29.638178 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:29.638182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:29.638252 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:29.638353 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:29.639150 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.639932 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.641117 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:29.641580 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:29.641709 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:29.645425 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:52159: BlockingRecv error: recv got EOF from 127.22.221.65:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 07:34:30 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:30 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:30.087181 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:30.078859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:41419)
0926 07:34:30.079416 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:30.079433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:30.079548 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:30.079823 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:30.079826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:30.080240 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:30.080425 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:30.080434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.081424 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.081428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:30.082291 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:30.082301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.082987 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.082992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:30.083166 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:30.083891 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:30.083909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:30.086944 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
W20260926 07:34:30.087458 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:30.079051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:41419 (local address 127.22.221.126:33337)
0926 07:34:30.079299 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:30.079306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:30.079509 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:34:30.079632 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:30.079635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:30.079697 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:30.079775 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:30.080560 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.081295 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.082423 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.082853 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.082952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:30.087163 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:41419: BlockingRecv error: recv got EOF from 127.22.221.66:41419 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 07:34:30 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:30 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:30.529002 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:30.520681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:57685)
0926 07:34:30.521150 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:30.521169 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:30.521325 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:30.521660 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:30.521665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:30.522093 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:30.522276 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:30.522285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.523400 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.523408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:30.524302 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:30.524314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.525012 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.525018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:30.525197 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:30.525813 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:30.525831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:30.528651 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
W20260926 07:34:30.529070 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:30.520727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:57685 (local address 127.22.221.126:33337)
0926 07:34:30.520928 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:30.520936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:30.521258 (+ 322us) server_negotiation.cc:407] Connection header received
0926 07:34:30.521466 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:30.521471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:30.521531 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:30.521630 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:30.522472 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.523219 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.524449 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.524863 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.524995 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:30.528838 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:57685: BlockingRecv error: recv got EOF from 127.22.221.67:57685 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:34:30 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:30 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:30.654918 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:30.646665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:48827)
0926 07:34:30.647323 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:30.647339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:30.647449 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:30.647735 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:30.647738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:30.648171 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:30.648379 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:30.648389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.649749 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.649753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:30.650625 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:30.650635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.651359 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.651363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:30.651481 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:30.652115 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:30.652132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:30.654686 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":330,"threads_started":1}
W20260926 07:34:30.655074 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:30.646758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:48827 (local address 127.22.221.126:33337)
0926 07:34:30.646937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:30.646942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:30.647414 (+ 472us) server_negotiation.cc:407] Connection header received
0926 07:34:30.647549 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:30.647552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:30.647613 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:30.647695 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:30.648493 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.649636 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.650735 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:30.651194 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:30.651310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:30.654867 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:48827: BlockingRecv error: recv got EOF from 127.22.221.65:48827 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 07:34:31 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:31 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:31.096426 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:31.088067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:59989)
0926 07:34:31.088574 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:31.088589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:31.088745 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:31.089078 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:31.089083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:31.089508 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:31.089658 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:31.089668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.090697 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.090704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:31.091683 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:31.091695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.092434 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.092440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:31.092639 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.093166 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:31.093182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:31.096079 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":308,"threads_started":1}
W20260926 07:34:31.096506 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:31.088106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:59989 (local address 127.22.221.126:33337)
0926 07:34:31.088336 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:31.088343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:31.088676 (+ 333us) server_negotiation.cc:407] Connection header received
0926 07:34:31.088870 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:31.088875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:31.088949 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:31.089049 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:31.089815 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.090554 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.091827 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.092286 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.092426 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.096286 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:59989: BlockingRecv error: recv got EOF from 127.22.221.66:59989 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 07:34:31 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:31 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:31.538517 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:31.530073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:48343)
0926 07:34:31.530793 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:31.530813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:31.530958 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:31.531367 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:31.531372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:31.531796 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:31.531952 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:31.531960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.533013 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.533019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:31.533948 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:31.533959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.534684 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.534690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:31.534882 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.535473 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:31.535491 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:31.538195 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":504,"threads_started":1}
W20260926 07:34:31.538658 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:31.530139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:48343 (local address 127.22.221.126:33337)
0926 07:34:31.530358 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:31.530363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:31.530898 (+ 535us) server_negotiation.cc:407] Connection header received
0926 07:34:31.531071 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:31.531075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:31.531180 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:31.531340 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:31.532091 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.532869 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.534094 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.534515 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.534673 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.538421 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:48343: BlockingRecv error: recv got EOF from 127.22.221.67:48343 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 07:34:31 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:31 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:31.664184 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:31.655838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:54367)
0926 07:34:31.656344 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:31.656361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:31.656474 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:31.656885 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:31.656888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:31.657262 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:31.657413 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:31.657420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.658458 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.658462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:31.659463 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:31.659474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.660462 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.660468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:31.660629 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.661171 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:31.661187 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:31.663838 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":284,"threads_started":1}
W20260926 07:34:31.664273 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:31.655965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:54367 (local address 127.22.221.126:33337)
0926 07:34:31.656223 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:31.656229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:31.656439 (+ 210us) server_negotiation.cc:407] Connection header received
0926 07:34:31.656569 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:31.656572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:31.656748 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:31.656825 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:31.657550 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.658329 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.659600 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.660255 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.660429 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.664024 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:54367: BlockingRecv error: recv got EOF from 127.22.221.65:54367 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 07:34:32 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:32 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:32.106143 32679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:32.097634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:42805)
0926 07:34:32.098139 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:32.098155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:32.098305 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:32.098759 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:32.098764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:32.099401 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:32.099607 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:32.099617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.100723 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.100727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:32.101703 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:32.101715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.102415 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.102420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:32.102531 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.103172 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:32.103190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:32.105914 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":247,"threads_started":1}
W20260926 07:34:32.106288 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:32.097749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:42805 (local address 127.22.221.126:33337)
0926 07:34:32.097965 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:32.097972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:32.098241 (+ 269us) server_negotiation.cc:407] Connection header received
0926 07:34:32.098515 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:32.098521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:32.098612 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:32.098725 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:32.099735 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.100608 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.101819 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.102259 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.102354 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.106088 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:42805: BlockingRecv error: recv got EOF from 127.22.221.66:42805 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 07:34:32 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:32 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:32.552546 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:32.539555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:49119)
0926 07:34:32.540056 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:32.540074 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:32.540225 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:32.540566 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:32.540571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:32.541018 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:32.541170 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:32.541179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.542180 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.542185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:32.543140 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:32.543151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.544096 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.544102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:32.544281 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.549341 (+ 5060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:32.549366 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:32.552271 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":249,"threads_started":1}
W20260926 07:34:32.552827 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:32.539614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:49119 (local address 127.22.221.126:33337)
0926 07:34:32.539825 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:32.539833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:32.540156 (+ 323us) server_negotiation.cc:407] Connection header received
0926 07:34:32.540362 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:32.540373 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:32.540435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:32.540538 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:32.541326 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.542042 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.543331 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.543935 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.544031 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.552523 (+ 8492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:49119: BlockingRecv error: recv got EOF from 127.22.221.67:49119 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 07:34:32 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:32 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:32.673378 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:32.665096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:60485)
0926 07:34:32.665596 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:32.665611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:32.665771 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:32.666164 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:32.666171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:32.666552 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:32.666733 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:32.666742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.667880 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.667885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:32.668798 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:32.668808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.669548 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.669554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:32.669727 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.670401 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:32.670418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:32.673139 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
W20260926 07:34:32.673574 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:32.665191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:60485 (local address 127.22.221.126:33337)
0926 07:34:32.665517 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:32.665523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:32.665697 (+ 174us) server_negotiation.cc:407] Connection header received
0926 07:34:32.665934 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:32.665939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:32.666020 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:32.666124 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:32.666898 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.667729 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.668955 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.669396 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.669523 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.673328 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:60485: BlockingRecv error: recv got EOF from 127.22.221.65:60485 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
Sep 26 07:34:33 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:33 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:33.114778 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:33.107065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:38107)
0926 07:34:33.107571 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:33.107587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:33.107700 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:33.107993 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:33.107996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:33.108357 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:33.108531 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:33.108539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.109497 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.109500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:33.110443 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:33.110456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.111146 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.111150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:33.111289 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:33.111867 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:33.111884 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:33.114542 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":292,"threads_started":1}
W20260926 07:34:33.114959 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:33.107163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:38107 (local address 127.22.221.126:33337)
0926 07:34:33.107346 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:33.107350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:33.107657 (+ 307us) server_negotiation.cc:407] Connection header received
0926 07:34:33.107791 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:33.107794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:33.107866 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:33.107945 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:33.108652 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.109376 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.110567 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.111028 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.111118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:33.114723 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:38107: BlockingRecv error: recv got EOF from 127.22.221.66:38107 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 07:34:33 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:33 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.67@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:33.562206 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:33.553655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.67:50263)
0926 07:34:33.554171 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:33.554193 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:33.554371 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:33.554747 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:33.554753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:33.555157 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:33.555375 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:33.555384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.556600 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.556604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:33.557795 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:33.557807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.558497 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.558501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:33.558619 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:33.559330 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:33.559349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:33.561975 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
W20260926 07:34:33.562429 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:33.553777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:50263 (local address 127.22.221.126:33337)
0926 07:34:33.554005 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:33.554013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:33.554286 (+ 273us) server_negotiation.cc:407] Connection header received
0926 07:34:33.554506 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:33.554511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:33.554601 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:33.554712 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:33.555531 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.556429 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.557927 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.558370 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.558466 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:33.562197 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.67:50263: BlockingRecv error: recv got EOF from 127.22.221.67:50263 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 07:34:33 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:33 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.65@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:33.683338 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:33.674872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.65:55665)
0926 07:34:33.675487 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:33.675504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:33.675617 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:33.675964 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:33.675967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:33.676494 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:33.676662 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:33.676671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.677970 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.677975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:33.678930 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:33.678941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.679626 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.679630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:33.679744 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:33.680323 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:33.680339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:33.683016 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":340,"threads_started":1}
W20260926 07:34:33.683491 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:33.674963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:55665 (local address 127.22.221.126:33337)
0926 07:34:33.675140 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:33.675145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:33.675581 (+ 436us) server_negotiation.cc:407] Connection header received
0926 07:34:33.675716 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:33.675720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:33.675816 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:33.675914 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:33.676780 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.677797 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.679055 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:33.679507 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:33.679600 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:33.683211 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.65:55665: BlockingRecv error: recv got EOF from 127.22.221.65:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 07:34:34 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:34:34 dist-test-slave-qlq8 krb5kdc[31640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.221.66@KRBTEST.COM for oryx/127.22.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:34:34.124506 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:34.116091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33337 (local address 127.22.221.66:36107)
0926 07:34:34.116585 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:34.116601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:34.116711 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:34.117013 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:34.117016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:34.117405 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:34.117568 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:34.117577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.118680 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.118685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:34.119896 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:34.119909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.120650 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.120654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:34.120764 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:34.121358 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:34.121376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:34.124101 (+ 2725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.221.126:33337: Server oryx/127.22.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
W20260926 07:34:34.124711 32376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:34.116144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:36107 (local address 127.22.221.126:33337)
0926 07:34:34.116369 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:34.116379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:34.116687 (+ 308us) server_negotiation.cc:407] Connection header received
0926 07:34:34.116815 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:34.116818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:34.116886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:34.116966 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:34.117694 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.118513 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.120025 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.120513 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.120607 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:34.124387 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.221.66:36107: BlockingRecv error: recv got EOF from 127.22.221.66:36107 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:34.390935 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32227
I20260926 07:34:34.403546 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:34.391990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.22.221.65:42359)
0926 07:34:34.392567 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:34.392574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:34.392611 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:34:34.392670 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:34.392677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:34.392936 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:34.393090 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:34.394532 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.395758 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.397018 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.397762 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.397869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:34.400297 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:34.400306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:34.400320 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:34.400351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:34.402998 (+ 2647us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.221.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 07:34:34.403314 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36324: Request ticket server kudu/127.22.221.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":294,"threads_started":1}
W20260926 07:34:34.403666 32687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36324: Request ticket server kudu/127.22.221.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 07:34:34.403777 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32227: Not authorized: Client connection negotiation failed: client connection to 127.22.221.65:42359: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.221.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:34:34.404016 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 32227
I20260926 07:34:34.420782 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32378
I20260926 07:34:34.433907 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:34.421840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.22.221.66:33253)
0926 07:34:34.422496 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:34.422502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:34.422547 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:34:34.422628 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:34.422635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:34.422860 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:34.423022 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:34.424410 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.425568 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.426955 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.427640 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.427759 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:34.430247 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:34.430255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:34.430269 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:34.430301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:34.433366 (+ 3065us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.221.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 07:34:34.433668 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43386: Request ticket server kudu/127.22.221.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":354,"threads_started":1}
W20260926 07:34:34.434026 32688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43386: Request ticket server kudu/127.22.221.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 07:34:34.434062 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32378: Not authorized: Client connection negotiation failed: client connection to 127.22.221.66:33253: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.221.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:34:34.434201 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 32378
I20260926 07:34:34.450687 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32509
I20260926 07:34:34.462378 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:34.451673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.22.221.67:46077)
0926 07:34:34.452259 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:34.452267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:34.452292 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:34:34.452366 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:34.452372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:34.452590 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:34.452769 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:34.453980 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.454953 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.456243 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:34.456792 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:34.456891 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:34.459407 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:34.459415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:34.459428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:34.459460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:34.461857 (+ 2397us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.221.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 07:34:34.462100 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38428: Request ticket server kudu/127.22.221.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
W20260926 07:34:34.462597 32689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38428: Request ticket server kudu/127.22.221.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 07:34:34.462646 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32509: Not authorized: Client connection negotiation failed: client connection to 127.22.221.67:46077: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.221.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:34:34.462818 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 32509
I20260926 07:34:34.482867 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32158
I20260926 07:34:34.484212 32220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:34.600342 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 32158
2026-09-26T07:34:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20934 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T07:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:34Z Disabled control of system clock
I20260926 07:34:34.674913 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:34175
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:34175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:34:34.999428 32694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:34.999735 32694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:34.999907 32694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:35.007423 32694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:35.007534 32694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:35.007645 32694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:35.007752 32694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:35.007835 32694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:35.017067 32694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:34175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:35.018373 32694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:35.020190 32694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:35.032727 32700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:35.032724 32702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:35.032866 32694 server_base.cc:1034] running on GCE node
W20260926 07:34:35.033540 32699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:35.034446 32694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:35.035904 32694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:35.037108 32694 hybrid_clock.cc:648] HybridClock initialized: now 1790408075037068 us; error 30 us; skew 500 ppm
I20260926 07:34:35.037425 32694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:35.039691 32694 webserver.cc:460] Webserver started at http://127.22.221.126:43905/ using document root <none> and password file <none>
I20260926 07:34:35.040386 32694 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:35.040472 32694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:35.040740 32694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:35.043659 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "5d1071d61d674357a340ffd0a1b4bef9"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-qlq8"
server_key: "53d1fbe010fd4b50d58659ecfc7772ef"
server_key_iv: "217a071f6e983390edc0c6ce52e61553"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.044409 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "5d1071d61d674357a340ffd0a1b4bef9"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-qlq8"
server_key: "53d1fbe010fd4b50d58659ecfc7772ef"
server_key_iv: "217a071f6e983390edc0c6ce52e61553"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.049307 32694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:34:35.052649 32708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:35.054081 32694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:34:35.054261 32694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "5d1071d61d674357a340ffd0a1b4bef9"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-qlq8"
server_key: "53d1fbe010fd4b50d58659ecfc7772ef"
server_key_iv: "217a071f6e983390edc0c6ce52e61553"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.054443 32694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:35.072647 32694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:35.073498 32694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:35.073745 32694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:35.092844 32694 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:34175
I20260926 07:34:35.092867 32759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:34175 every 8 connection(s)
I20260926 07:34:35.094282 32694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:35.094753 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 32694
I20260926 07:34:35.094959 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:34:35.095866 23413 external_mini_cluster.cc:1428] Setting key 79fbd1ca3ad7617affac73c6d65d58c5
I20260926 07:34:35.098974 32760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:35.109141 32763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:35.097986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.22.221.126:34175)
0926 07:34:35.098983 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:35.098993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:35.099032 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:34:35.100661 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:35.100681 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:35.101103 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:35.101499 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:35.102107 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.103439 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.104775 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.105420 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.105645 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:35.106845 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:35.106852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:35.106865 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:35.107474 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:35.107555 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:35.107578 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:35.107716 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:35.107945 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 07:34:35.108249 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":366,"threads_started":1}
I20260926 07:34:35.110771 32760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9: Bootstrap starting.
I20260926 07:34:35.114104 32760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:35.115135 32760 log.cc:824] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:35.118144 32760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9: No bootstrap required, opened a new log
I20260926 07:34:35.125829 32760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1071d61d674357a340ffd0a1b4bef9" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34175 } }
I20260926 07:34:35.126223 32760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:35.126381 32760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1071d61d674357a340ffd0a1b4bef9, State: Initialized, Role: FOLLOWER
I20260926 07:34:35.126976 32760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1071d61d674357a340ffd0a1b4bef9" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34175 } }
I20260926 07:34:35.127221 32760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:35.127404 32760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:35.127600 32760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:35.129871 32760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1071d61d674357a340ffd0a1b4bef9" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34175 } }
I20260926 07:34:35.130390 32760 leader_election.cc:302] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d1071d61d674357a340ffd0a1b4bef9; no voters:
I20260926 07:34:35.130993 32760 leader_election.cc:288] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:35.131098 32765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:35.131949 32765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1071d61d674357a340ffd0a1b4bef9, State: Running, Role: LEADER
I20260926 07:34:35.132485 32765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1071d61d674357a340ffd0a1b4bef9" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34175 } }
I20260926 07:34:35.133230 32760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:35.135186 32767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d1071d61d674357a340ffd0a1b4bef9. Latest consensus state: current_term: 1 leader_uuid: "5d1071d61d674357a340ffd0a1b4bef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1071d61d674357a340ffd0a1b4bef9" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34175 } } }
I20260926 07:34:35.135210 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d1071d61d674357a340ffd0a1b4bef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1071d61d674357a340ffd0a1b4bef9" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34175 } } }
I20260926 07:34:35.135707 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:35.136292 32767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:35.137617 302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:35.142817 302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:35.151734 302 catalog_manager.cc:1347] Generated new cluster ID: 2cf5b9b815fe49dfa4a0ec04141677dd
I20260926 07:34:35.151860 302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:35.181437 302 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:35.182713 302 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:35.197312 302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d1071d61d674357a340ffd0a1b4bef9: Generated new TSK 0
I20260926 07:34:35.198169 302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:35.209293 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:34175
--builtin_ntp_servers=127.22.221.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:34:35.530934 316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:35.531200 316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:35.531308 316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:35.538785 316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:35.538915 316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:35.539052 316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:35.547981 316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:34175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:35.549136 316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:35.550817 316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:35.563530 324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:35.563562 322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:35.563591 321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:35.565124 316 server_base.cc:1034] running on GCE node
I20260926 07:34:35.565765 316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:35.566653 316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:35.567886 316 hybrid_clock.cc:648] HybridClock initialized: now 1790408075567845 us; error 56 us; skew 500 ppm
I20260926 07:34:35.568199 316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:35.570284 316 webserver.cc:460] Webserver started at http://127.22.221.65:35659/ using document root <none> and password file <none>
I20260926 07:34:35.570955 316 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:35.571089 316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:35.571460 316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:35.574147 316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "ab8e2c6d88334577ae4ee14d8b2ba8f8"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-qlq8"
server_key: "fbd597d4e04de620100d9dd72e2f7189"
server_key_iv: "bd2cffd6c83e3bee8e8a68db48b059df"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.574939 316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "ab8e2c6d88334577ae4ee14d8b2ba8f8"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-qlq8"
server_key: "fbd597d4e04de620100d9dd72e2f7189"
server_key_iv: "bd2cffd6c83e3bee8e8a68db48b059df"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.579728 316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 07:34:35.582868 330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:35.584208 316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:35.584337 316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "ab8e2c6d88334577ae4ee14d8b2ba8f8"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-qlq8"
server_key: "fbd597d4e04de620100d9dd72e2f7189"
server_key_iv: "bd2cffd6c83e3bee8e8a68db48b059df"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.584506 316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:35.608461 316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:35.609301 316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:35.609575 316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:35.610463 316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:35.611902 316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:35.612023 316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:35.612151 316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:35.612241 316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:35.644748 316 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:39993
I20260926 07:34:35.644834 442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:39993 every 8 connection(s)
I20260926 07:34:35.646082 316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:35.654644 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 316
I20260926 07:34:35.654842 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:35.655584 23413 external_mini_cluster.cc:1428] Setting key d1ffbdfeca67cc0a3a27b7fd04055ba3
I20260926 07:34:35.658301 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:35.648551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:35569 (local address 127.22.221.126:34175)
0926 07:34:35.648992 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:35.648996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:35.650050 (+ 1054us) server_negotiation.cc:407] Connection header received
0926 07:34:35.651861 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:35.651867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:35.651955 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:35.652071 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:35.653100 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.653800 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.655491 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.655936 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.656064 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:35.657602 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:35.657606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:35.657609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:35.657624 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:35.657659 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:35.657670 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:35.657733 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:35.657999 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 07:34:35.658068 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":260,"threads_started":1}
I20260926 07:34:35.658633 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:34175
--builtin_ntp_servers=127.22.221.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:34:35.659744 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:35.649053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:34175 (local address 127.22.221.65:35569)
0926 07:34:35.649906 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:35.649940 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:35.651600 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:35.652289 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:35.652297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:35.652386 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:35.652936 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:35.652953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.653964 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.653968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:35.655353 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:35.655365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.656158 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.656165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:35.656487 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:35.657252 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:35.657255 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:35.657304 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:35.657391 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:35.657768 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:35.657781 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:35.657790 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:34:35.657993 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 07:34:35.658370 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":252,"threads_started":1}
I20260926 07:34:35.661170 443 heartbeater.cc:347] Connected to a master server at 127.22.221.126:34175
I20260926 07:34:35.661460 443 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:35.662272 443 heartbeater.cc:511] Master 127.22.221.126:34175 requested a full tablet report, sending...
I20260926 07:34:35.664273 32725 ts_manager.cc:194] Registered new tserver with Master: ab8e2c6d88334577ae4ee14d8b2ba8f8 (127.22.221.65:39993)
I20260926 07:34:35.666090 32725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.65:35569
W20260926 07:34:35.974721 447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:35.974982 447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:35.975062 447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:35.982143 447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:35.982232 447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:35.982336 447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:35.991221 447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:34175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:35.992460 447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:35.994128 447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:36.006873 453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:36.006928 452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:36.006888 455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:36.007908 447 server_base.cc:1034] running on GCE node
I20260926 07:34:36.008473 447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:36.009389 447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:36.010632 447 hybrid_clock.cc:648] HybridClock initialized: now 1790408076010586 us; error 43 us; skew 500 ppm
I20260926 07:34:36.010941 447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:36.013104 447 webserver.cc:460] Webserver started at http://127.22.221.66:44457/ using document root <none> and password file <none>
I20260926 07:34:36.013788 447 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:36.013877 447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:36.014217 447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:36.016959 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "562e3a6c909943b199419d720e90fc89"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-qlq8"
server_key: "73a7da0a534991714975fb4299202075"
server_key_iv: "6262cf36f117854559d3a7df18c6cec3"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.017733 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "562e3a6c909943b199419d720e90fc89"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-qlq8"
server_key: "73a7da0a534991714975fb4299202075"
server_key_iv: "6262cf36f117854559d3a7df18c6cec3"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.022680 447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:34:36.025954 461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.027110 447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:34:36.027323 447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "562e3a6c909943b199419d720e90fc89"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-qlq8"
server_key: "73a7da0a534991714975fb4299202075"
server_key_iv: "6262cf36f117854559d3a7df18c6cec3"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.027537 447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:36.075925 447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:36.076807 447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:36.077118 447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:36.078082 447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:36.079507 447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:36.079629 447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.079763 447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:36.079854 447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.113670 447 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:32891
I20260926 07:34:36.113716 573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:32891 every 8 connection(s)
I20260926 07:34:36.114969 447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:36.121424 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 447
I20260926 07:34:36.121595 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:36.122046 23413 external_mini_cluster.cc:1428] Setting key 598df0207963bb5b635fd168b30a0a5f
I20260926 07:34:36.124873 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:34175
--builtin_ntp_servers=127.22.221.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:34:36.127888 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.117507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:49559 (local address 127.22.221.126:34175)
0926 07:34:36.117732 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.117738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.119116 (+ 1378us) server_negotiation.cc:407] Connection header received
0926 07:34:36.120863 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.120868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.120933 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.121046 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.122211 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.123100 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.124797 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.125503 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.125653 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.126955 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.126964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.126971 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.126994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.127036 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.127052 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.127143 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.127398 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.127504 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 07:34:36.129037 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.118072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:34175 (local address 127.22.221.66:49559)
0926 07:34:36.118961 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:36.118996 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:36.120626 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:36.121236 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:36.121243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:36.121341 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:36.122009 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:36.122026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.123296 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.123300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:36.124643 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:36.124659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.125713 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.125718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:36.125931 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.126647 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:36.126650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:36.126689 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:36.126761 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:36.127149 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:36.127157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:36.127162 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:34:36.127369 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:34:36.127667 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":269,"threads_started":1}
I20260926 07:34:36.130442 574 heartbeater.cc:347] Connected to a master server at 127.22.221.126:34175
I20260926 07:34:36.130726 574 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:36.131618 574 heartbeater.cc:511] Master 127.22.221.126:34175 requested a full tablet report, sending...
I20260926 07:34:36.133198 32725 ts_manager.cc:194] Registered new tserver with Master: 562e3a6c909943b199419d720e90fc89 (127.22.221.66:32891)
I20260926 07:34:36.134112 32725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.66:49559
W20260926 07:34:36.442282 577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:36.442536 577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:36.442703 577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:36.449747 577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:36.449873 577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:36.450062 577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:36.459064 577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:34175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:36.460320 577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:36.462105 577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:36.477803 585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:36.477834 582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:36.477851 583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:36.483343 577 server_base.cc:1034] running on GCE node
I20260926 07:34:36.484277 577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:36.485268 577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:36.486512 577 hybrid_clock.cc:648] HybridClock initialized: now 1790408076486461 us; error 49 us; skew 500 ppm
I20260926 07:34:36.486878 577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:36.489283 577 webserver.cc:460] Webserver started at http://127.22.221.67:45097/ using document root <none> and password file <none>
I20260926 07:34:36.489995 577 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:36.490130 577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:36.490458 577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:36.493311 577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "1cfa31698ab44e79941bb2dff1a672a3"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-qlq8"
server_key: "b336bef347161ec5080750377d9b4fa6"
server_key_iv: "642e1914c89053aafbc2caaa773fcc96"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.494117 577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "1cfa31698ab44e79941bb2dff1a672a3"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-qlq8"
server_key: "b336bef347161ec5080750377d9b4fa6"
server_key_iv: "642e1914c89053aafbc2caaa773fcc96"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.499027 577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 07:34:36.502453 591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.503754 577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:36.503937 577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "1cfa31698ab44e79941bb2dff1a672a3"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-qlq8"
server_key: "b336bef347161ec5080750377d9b4fa6"
server_key_iv: "642e1914c89053aafbc2caaa773fcc96"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.504155 577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:36.520788 577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:36.521612 577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:36.521914 577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:36.522799 577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:36.524219 577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:36.524360 577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.524492 577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:36.524581 577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.557449 577 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:39577
I20260926 07:34:36.557480 703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:39577 every 8 connection(s)
I20260926 07:34:36.558857 577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:36.568641 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 577
I20260926 07:34:36.568846 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:36.569484 23413 external_mini_cluster.cc:1428] Setting key 991c94d96d3c34ef222d7a1d57b1658c
I20260926 07:34:36.571487 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.561506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:33397 (local address 127.22.221.126:34175)
0926 07:34:36.561678 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.561683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.562999 (+ 1316us) server_negotiation.cc:407] Connection header received
0926 07:34:36.564748 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.564753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.564818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.564905 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.565862 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.566569 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.568087 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.568532 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.568682 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.570574 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.570579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.570582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.570599 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.570630 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.570642 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.570751 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.570992 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.571083 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:34:36.572422 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.562034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:34175 (local address 127.22.221.67:33397)
0926 07:34:36.562878 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:36.562911 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:36.564508 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:36.565098 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:36.565105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:36.565190 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:36.565725 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:36.565741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.566686 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.566689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:36.567966 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:36.567977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.568854 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.568891 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:36.569375 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.570214 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:36.570219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:36.570275 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:36.570367 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:36.570766 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:36.570777 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:36.570785 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:34:36.570980 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 07:34:36.571394 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":266,"threads_started":1}
I20260926 07:34:36.573663 704 heartbeater.cc:347] Connected to a master server at 127.22.221.126:34175
I20260926 07:34:36.573956 704 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:36.574673 704 heartbeater.cc:511] Master 127.22.221.126:34175 requested a full tablet report, sending...
I20260926 07:34:36.576015 32725 ts_manager.cc:194] Registered new tserver with Master: 1cfa31698ab44e79941bb2dff1a672a3 (127.22.221.67:39577)
I20260926 07:34:36.576778 32725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.67:33397
I20260926 07:34:36.584879 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:34:36.591111 445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:36.589514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.22.221.126:34175)
0926 07:34:36.589670 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.589674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.590138 (+ 464us) server_negotiation.cc:407] Connection header received
0926 07:34:36.590341 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.590348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.590423 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.590529 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.590947 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47914: BlockingRecv error: recv got EOF from 127.0.0.1:47914 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:34:36.592042 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 316
I20260926 07:34:36.599385 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.592930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.22.221.65:39993)
0926 07:34:36.593495 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.593502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.593526 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:34:36.593578 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.593584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.593776 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.593898 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.594204 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.595168 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.596408 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.596938 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.597037 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.597997 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.598004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.598016 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.598574 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.598616 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.598634 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.598730 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.598856 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.598982 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":305,"threads_started":1}
I20260926 07:34:36.600530 438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:36.680639 443 heartbeater.cc:503] Master 127.22.221.126:34175 was elected leader, sending a full tablet report...
I20260926 07:34:36.682231 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 316
I20260926 07:34:36.698493 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 447
I20260926 07:34:36.705896 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.699453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.22.221.66:32891)
0926 07:34:36.699958 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.699965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.699991 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:34:36.700050 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.700056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.700265 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.700406 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.700736 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.701729 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.702857 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.703471 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.703570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.704548 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.704554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.704566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.705136 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.705179 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.705196 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.705282 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.705408 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.705550 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":259,"threads_started":1}
I20260926 07:34:36.707068 569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:36.788610 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 447
I20260926 07:34:36.804193 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 577
I20260926 07:34:36.812409 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.805126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.22.221.67:39577)
0926 07:34:36.805745 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.805751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.805780 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:34:36.805843 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.805849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.806058 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.806236 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.806697 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.807809 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.809094 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.809647 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.809782 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.810868 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.810877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.810889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.811564 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.811612 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.811631 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.811755 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.811918 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.812063 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":332,"threads_started":1}
I20260926 07:34:36.813602 699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:36.895648 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 577
I20260926 07:34:36.911588 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 32694
I20260926 07:34:36.912865 32755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:37.018246 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 32694
2026-09-26T07:34:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2414 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T07:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:37Z Disabled control of system clock
I20260926 07:34:37.099495 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42697
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:34:37.417701 723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:37.417968 723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:37.418081 723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:37.425112 723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:37.425210 723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:37.425295 723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:37.425405 723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:37.425479 723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:37.434792 723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:42697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:37.436004 723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:37.437716 723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:37.449133 729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:37.449199 728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:37.449221 731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:37.449854 723 server_base.cc:1034] running on GCE node
I20260926 07:34:37.450500 723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:37.451856 723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:37.453060 723 hybrid_clock.cc:648] HybridClock initialized: now 1790408077453018 us; error 27 us; skew 500 ppm
I20260926 07:34:37.453394 723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:37.455679 723 webserver.cc:460] Webserver started at http://127.22.221.126:33783/ using document root <none> and password file <none>
I20260926 07:34:37.456337 723 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:37.456449 723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:37.456766 723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:37.459539 723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "2f204f014a534e0ab65ba0acdea55e29"
format_stamp: "Formatted at 2026-09-26 07:34:37 on dist-test-slave-qlq8"
server_key: "2befce8c4f64c36eef68e526015fb425"
server_key_iv: "8f4777c5f949b0eb0a07cd955ec36bae"
server_key_version: "encryptionkey@0"
I20260926 07:34:37.460496 723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "2f204f014a534e0ab65ba0acdea55e29"
format_stamp: "Formatted at 2026-09-26 07:34:37 on dist-test-slave-qlq8"
server_key: "2befce8c4f64c36eef68e526015fb425"
server_key_iv: "8f4777c5f949b0eb0a07cd955ec36bae"
server_key_version: "encryptionkey@0"
I20260926 07:34:37.465479 723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:34:37.468703 737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:37.469954 723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:34:37.470124 723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "2f204f014a534e0ab65ba0acdea55e29"
format_stamp: "Formatted at 2026-09-26 07:34:37 on dist-test-slave-qlq8"
server_key: "2befce8c4f64c36eef68e526015fb425"
server_key_iv: "8f4777c5f949b0eb0a07cd955ec36bae"
server_key_version: "encryptionkey@0"
I20260926 07:34:37.470326 723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:37.475814 723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:37.476506 723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:37.476786 723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:37.495144 723 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:42697
I20260926 07:34:37.495182 788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:42697 every 8 connection(s)
I20260926 07:34:37.496438 723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:37.499975 789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:37.505888 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 723
I20260926 07:34:37.506107 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:34:37.506719 23413 external_mini_cluster.cc:1428] Setting key 01c5e4a6654ee944c542cf0c2b759e0f
I20260926 07:34:37.510557 789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29: Bootstrap starting.
I20260926 07:34:37.514438 789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:37.515767 789 log.cc:824] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:37.515946 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:37.508694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.22.221.126:42697)
0926 07:34:37.509710 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:37.509723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:37.509772 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:34:37.512074 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:37.512109 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:37.512670 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:37.513084 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:37.514041 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:37.514050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:37.514065 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:37.514081 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:37.514151 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:37.514175 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:37.514364 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:37.514506 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 07:34:37.514771 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":291,"threads_started":1}
I20260926 07:34:37.518569 789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29: No bootstrap required, opened a new log
I20260926 07:34:37.525534 789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f204f014a534e0ab65ba0acdea55e29" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42697 } }
I20260926 07:34:37.525822 789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:37.525884 789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f204f014a534e0ab65ba0acdea55e29, State: Initialized, Role: FOLLOWER
I20260926 07:34:37.526317 789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f204f014a534e0ab65ba0acdea55e29" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42697 } }
I20260926 07:34:37.526499 789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:37.526588 789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:37.526701 789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:37.528744 789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f204f014a534e0ab65ba0acdea55e29" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42697 } }
I20260926 07:34:37.529181 789 leader_election.cc:302] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f204f014a534e0ab65ba0acdea55e29; no voters:
I20260926 07:34:37.529870 789 leader_election.cc:288] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:37.529968 794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:37.530661 794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [term 1 LEADER]: Becoming Leader. State: Replica: 2f204f014a534e0ab65ba0acdea55e29, State: Running, Role: LEADER
I20260926 07:34:37.531260 794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f204f014a534e0ab65ba0acdea55e29" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42697 } }
I20260926 07:34:37.531982 789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:37.533864 795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f204f014a534e0ab65ba0acdea55e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f204f014a534e0ab65ba0acdea55e29" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42697 } } }
I20260926 07:34:37.534241 795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:37.536125 796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f204f014a534e0ab65ba0acdea55e29. Latest consensus state: current_term: 1 leader_uuid: "2f204f014a534e0ab65ba0acdea55e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f204f014a534e0ab65ba0acdea55e29" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 42697 } } }
I20260926 07:34:37.536725 796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:37.537753 802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:37.542783 802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:37.550545 802 catalog_manager.cc:1347] Generated new cluster ID: 3c9d658b5495461ca683c4c57c12267b
I20260926 07:34:37.550657 802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:37.574990 802 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:37.575449 802 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:37.583328 802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f204f014a534e0ab65ba0acdea55e29: Generated new TSK 0
I20260926 07:34:37.584023 802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:37.589298 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42697
--builtin_ntp_servers=127.22.221.84:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:34:37.941823 813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:37.942080 813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:37.942178 813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:37.949379 813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:37.949478 813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:37.949590 813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:37.958738 813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:37.960094 813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:37.961898 813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:37.974874 818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:37.974907 819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:37.974931 821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:37.975808 813 server_base.cc:1034] running on GCE node
I20260926 07:34:37.976428 813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:37.977375 813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:37.978595 813 hybrid_clock.cc:648] HybridClock initialized: now 1790408077978555 us; error 63 us; skew 500 ppm
I20260926 07:34:37.978922 813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:37.986665 813 webserver.cc:460] Webserver started at http://127.22.221.65:43291/ using document root <none> and password file <none>
I20260926 07:34:37.987452 813 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:37.987581 813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:37.987906 813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:37.992471 813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "8419748f498349d797b1286711e44410"
format_stamp: "Formatted at 2026-09-26 07:34:37 on dist-test-slave-qlq8"
server_key: "f5e07c959cd396f601ec1b3ce102b287"
server_key_iv: "603227bda62cb12aad7d6db13f721541"
server_key_version: "encryptionkey@0"
I20260926 07:34:37.993268 813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "8419748f498349d797b1286711e44410"
format_stamp: "Formatted at 2026-09-26 07:34:37 on dist-test-slave-qlq8"
server_key: "f5e07c959cd396f601ec1b3ce102b287"
server_key_iv: "603227bda62cb12aad7d6db13f721541"
server_key_version: "encryptionkey@0"
I20260926 07:34:37.998245 813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 07:34:38.001678 827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.003175 813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:38.003378 813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "8419748f498349d797b1286711e44410"
format_stamp: "Formatted at 2026-09-26 07:34:37 on dist-test-slave-qlq8"
server_key: "f5e07c959cd396f601ec1b3ce102b287"
server_key_iv: "603227bda62cb12aad7d6db13f721541"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.003605 813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:38.009289 813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:38.010105 813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:38.010391 813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:38.011332 813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:38.012770 813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:38.012882 813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.013005 813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:38.013103 813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.045374 813 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:35071
I20260926 07:34:38.045430 939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:35071 every 8 connection(s)
I20260926 07:34:38.046705 813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:38.053802 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 813
I20260926 07:34:38.054029 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:38.054843 23413 external_mini_cluster.cc:1428] Setting key dfca56bfb6f9bcdc2bc63116cb2898ad
I20260926 07:34:38.055048 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.049368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:58283 (local address 127.22.221.126:42697)
0926 07:34:38.049787 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.049791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.051004 (+ 1213us) server_negotiation.cc:407] Connection header received
0926 07:34:38.052671 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.052676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.052728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.052828 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.054271 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:38.054276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:38.054279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:38.054295 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:38.054342 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:38.054353 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:38.054449 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:38.054727 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 07:34:38.054774 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
I20260926 07:34:38.056166 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.049993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42697 (local address 127.22.221.65:58283)
0926 07:34:38.050853 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:38.050892 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:38.052416 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:38.053070 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:38.053078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:38.053170 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:38.053914 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:38.053919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:38.053982 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:38.054102 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:38.054454 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:38.054464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:38.054474 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:34:38.054821 (+ 347us) client_negotiation.cc:240] Negotiation successful
0926 07:34:38.055084 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":249,"threads_started":1}
I20260926 07:34:38.057646 940 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42697
I20260926 07:34:38.057928 940 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:38.058053 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42697
--builtin_ntp_servers=127.22.221.84:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:34:38.058533 940 heartbeater.cc:511] Master 127.22.221.126:42697 requested a full tablet report, sending...
I20260926 07:34:38.060672 754 ts_manager.cc:194] Registered new tserver with Master: 8419748f498349d797b1286711e44410 (127.22.221.65:35071)
W20260926 07:34:38.374491 944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:38.374795 944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:38.374953 944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:38.381990 944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:38.382117 944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:38.382292 944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:38.391192 944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:38.392452 944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:38.394204 944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:38.407193 949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:38.407280 952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:38.407248 950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:38.408269 944 server_base.cc:1034] running on GCE node
I20260926 07:34:38.408839 944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:38.409724 944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:38.410945 944 hybrid_clock.cc:648] HybridClock initialized: now 1790408078410887 us; error 51 us; skew 500 ppm
I20260926 07:34:38.411298 944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:38.413455 944 webserver.cc:460] Webserver started at http://127.22.221.66:37755/ using document root <none> and password file <none>
I20260926 07:34:38.414144 944 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:38.414263 944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:38.414584 944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:38.417335 944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "5b29325f26cb470fac0cf9c6f2037acf"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-qlq8"
server_key: "d91fb9f4b2434107245a3445b7483d02"
server_key_iv: "e34b004f7302ae399d3de81ea4933095"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.418113 944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "5b29325f26cb470fac0cf9c6f2037acf"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-qlq8"
server_key: "d91fb9f4b2434107245a3445b7483d02"
server_key_iv: "e34b004f7302ae399d3de81ea4933095"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.422938 944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 07:34:38.426208 958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.427580 944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:38.427750 944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "5b29325f26cb470fac0cf9c6f2037acf"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-qlq8"
server_key: "d91fb9f4b2434107245a3445b7483d02"
server_key_iv: "e34b004f7302ae399d3de81ea4933095"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.427985 944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:38.433646 944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:38.434405 944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:38.434669 944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:38.435581 944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:38.436944 944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:38.437060 944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.437181 944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:38.437268 944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.471534 944 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:35821
I20260926 07:34:38.471613 1070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:35821 every 8 connection(s)
I20260926 07:34:38.473114 944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:38.480202 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 944
I20260926 07:34:38.480407 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:38.480859 23413 external_mini_cluster.cc:1428] Setting key f33593de98696b2d0e701e6f9d621728
I20260926 07:34:38.480873 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.475712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:48137 (local address 127.22.221.126:42697)
0926 07:34:38.475887 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.475891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.477203 (+ 1312us) server_negotiation.cc:407] Connection header received
0926 07:34:38.478766 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.478770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.478826 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.478910 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.480111 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:38.480118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:38.480123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:38.480145 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:38.480190 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:38.480206 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:38.480358 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:38.480573 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:34:38.480635 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:38.482307 1073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.476182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42697 (local address 127.22.221.66:48137)
0926 07:34:38.477061 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:38.477099 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:38.478536 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:38.479071 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:38.479078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:38.479163 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:38.479806 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:38.479809 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:38.479845 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:38.479925 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:38.480323 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:38.480333 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:38.480343 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:34:38.481148 (+ 805us) client_negotiation.cc:240] Negotiation successful
0926 07:34:38.481347 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":270,"threads_started":1}
I20260926 07:34:38.483594 1071 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42697
I20260926 07:34:38.483635 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42697
--builtin_ntp_servers=127.22.221.84:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:34:38.483896 1071 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:38.484431 1071 heartbeater.cc:511] Master 127.22.221.126:42697 requested a full tablet report, sending...
I20260926 07:34:38.485745 754 ts_manager.cc:194] Registered new tserver with Master: 5b29325f26cb470fac0cf9c6f2037acf (127.22.221.66:35821)
W20260926 07:34:38.803874 1074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:38.804147 1074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:38.804292 1074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:38.811551 1074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:38.811648 1074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:38.811833 1074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:38.820731 1074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:38.822007 1074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:38.823781 1074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:38.836746 1080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:38.836984 1079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:38.836802 1082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:38.837661 1074 server_base.cc:1034] running on GCE node
I20260926 07:34:38.838308 1074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:38.839283 1074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:38.840507 1074 hybrid_clock.cc:648] HybridClock initialized: now 1790408078840461 us; error 41 us; skew 500 ppm
I20260926 07:34:38.840845 1074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:34:38.842985 1074 webserver.cc:460] Webserver started at http://127.22.221.67:40833/ using document root <none> and password file <none>
I20260926 07:34:38.843705 1074 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:38.843842 1074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:38.844163 1074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:38.847131 1074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "b017219795304df49cd8547cc1719764"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-qlq8"
server_key: "2632dc99014a3f91961245eb623f3b7a"
server_key_iv: "56fb14ed0d7394e6b991360f489a33ce"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.847931 1074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "b017219795304df49cd8547cc1719764"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-qlq8"
server_key: "2632dc99014a3f91961245eb623f3b7a"
server_key_iv: "56fb14ed0d7394e6b991360f489a33ce"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.852772 1074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:34:38.856056 1088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.857307 1074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:34:38.857467 1074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "b017219795304df49cd8547cc1719764"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-qlq8"
server_key: "2632dc99014a3f91961245eb623f3b7a"
server_key_iv: "56fb14ed0d7394e6b991360f489a33ce"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.857679 1074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:38.863044 1074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:38.863858 1074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:38.864117 1074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:38.864943 1074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:38.866236 1074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:38.866361 1074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.866488 1074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:38.866573 1074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.899411 1074 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:44243
I20260926 07:34:38.899499 1200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:44243 every 8 connection(s)
I20260926 07:34:38.900775 1074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:38.906143 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 1074
I20260926 07:34:38.906338 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:38.906754 23413 external_mini_cluster.cc:1428] Setting key 0c18f6b32b6015bbbc386fc148151150
I20260926 07:34:38.909533 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.903377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:35627 (local address 127.22.221.126:42697)
0926 07:34:38.903553 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.903557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.904959 (+ 1402us) server_negotiation.cc:407] Connection header received
0926 07:34:38.906768 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.906773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.906875 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.907009 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.908727 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:38.908733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:38.908738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:38.908760 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:38.908806 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:38.908824 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:38.908948 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:38.909173 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 07:34:38.909259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:34:38.910475 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.903902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:42697 (local address 127.22.221.67:35627)
0926 07:34:38.904788 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:38.904826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:38.906478 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:38.907272 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:38.907284 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:38.907406 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:38.908324 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:38.908328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:38.908405 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:38.908525 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:38.908934 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:38.908946 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:38.908954 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:34:38.909219 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 07:34:38.909421 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260926 07:34:38.911839 1201 heartbeater.cc:347] Connected to a master server at 127.22.221.126:42697
I20260926 07:34:38.912124 1201 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:38.912700 1201 heartbeater.cc:511] Master 127.22.221.126:42697 requested a full tablet report, sending...
I20260926 07:34:38.913956 754 ts_manager.cc:194] Registered new tserver with Master: b017219795304df49cd8547cc1719764 (127.22.221.67:44243)
I20260926 07:34:38.922387 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:34:38.928368 942 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:38.926768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.22.221.126:42697)
0926 07:34:38.926971 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.926980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.927508 (+ 528us) server_negotiation.cc:407] Connection header received
0926 07:34:38.927638 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.927665 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.927717 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.927796 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.928128 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45842: BlockingRecv error: recv got EOF from 127.0.0.1:45842 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:34:38.929242 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 813
I20260926 07:34:38.932909 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.930117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.22.221.65:35071)
0926 07:34:38.930618 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.930625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.930649 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:34:38.930705 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.930711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.930924 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.931083 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.932193 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:38.932199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:38.932210 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:38.932226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:38.932263 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:38.932276 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:38.932413 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:38.932527 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 07:34:38.932643 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":206,"threads_started":1}
I20260926 07:34:38.934186 935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:39.015302 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 813
I20260926 07:34:39.031771 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 944
I20260926 07:34:39.035336 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.032684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.22.221.66:35821)
0926 07:34:39.033247 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:39.033255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:39.033294 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:34:39.033356 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:39.033361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:39.033572 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:39.033694 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:39.034627 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:39.034632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:39.034644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:39.034654 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:39.034693 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:39.034706 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:39.034815 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:39.034905 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:34:39.034987 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
I20260926 07:34:39.036487 1066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:39.117456 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 944
I20260926 07:34:39.134105 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 1074
I20260926 07:34:39.137944 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.135099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.22.221.67:44243)
0926 07:34:39.135791 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:39.135798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:39.135828 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:34:39.135889 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:39.135896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:39.136113 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:39.136259 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:39.137233 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:39.137240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:39.137252 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:39.137268 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:39.137307 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:39.137321 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:39.137465 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:39.137523 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 07:34:39.137624 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":381,"threads_started":1}
I20260926 07:34:39.139281 1196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:39.218765 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 1074
I20260926 07:34:39.234776 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 723
I20260926 07:34:39.235958 784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:39.339993 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 723
2026-09-26T07:34:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2320 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:39 dist-test-slave-qlq8 krb5kdc[1219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:39 dist-test-slave-qlq8 krb5kdc[1219](info): set up 2 sockets
Sep 26 07:34:39 dist-test-slave-qlq8 krb5kdc[1219](info): commencing operation
krb5kdc: starting...
W20260926 07:34:40.476516 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:34:40 dist-test-slave-qlq8 krb5kdc[1219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:34:40.694980 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37931
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:41.007268 1232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:41.007613 1232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:41.007777 1232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:41.015643 1232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:41.015765 1232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:41.015878 1232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:41.015969 1232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:41.016047 1232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:41.025827 1232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:41.027137 1232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:41.028434 1232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:41.040143 1240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:41.040344 1238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:41.040524 1237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:41.041363 1232 server_base.cc:1034] running on GCE node
I20260926 07:34:41.042143 1232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:41.043566 1232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:41.044799 1232 hybrid_clock.cc:648] HybridClock initialized: now 1790408081044760 us; error 49 us; skew 500 ppm
I20260926 07:34:41.045145 1232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:41 dist-test-slave-qlq8 krb5kdc[1219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408081, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:41.048249 1232 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:34:41.049672 1232 webserver.cc:460] Webserver started at http://127.22.221.126:36815/ using document root <none> and password file <none>
I20260926 07:34:41.050303 1232 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:41.050400 1232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:41.050652 1232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:41.053610 1232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "3916ff84d3f84c99bf24d30463d7af59"
format_stamp: "Formatted at 2026-09-26 07:34:41 on dist-test-slave-qlq8"
server_key: "f60e0319829839376585e52f73b83ffa"
server_key_iv: "8085bdfaa72c48583bf572e51f8f4f43"
server_key_version: "encryptionkey@0"
I20260926 07:34:41.054307 1232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "3916ff84d3f84c99bf24d30463d7af59"
format_stamp: "Formatted at 2026-09-26 07:34:41 on dist-test-slave-qlq8"
server_key: "f60e0319829839376585e52f73b83ffa"
server_key_iv: "8085bdfaa72c48583bf572e51f8f4f43"
server_key_version: "encryptionkey@0"
I20260926 07:34:41.059371 1232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260926 07:34:41.062608 1247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:41.064004 1232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:34:41.064188 1232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "3916ff84d3f84c99bf24d30463d7af59"
format_stamp: "Formatted at 2026-09-26 07:34:41 on dist-test-slave-qlq8"
server_key: "f60e0319829839376585e52f73b83ffa"
server_key_iv: "8085bdfaa72c48583bf572e51f8f4f43"
server_key_version: "encryptionkey@0"
I20260926 07:34:41.064392 1232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:41.088058 1232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:41.091990 1232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:41.092280 1232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:41.110073 1232 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37931
I20260926 07:34:41.110118 1298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37931 every 8 connection(s)
I20260926 07:34:41.111377 1232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:41.113936 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 1232
I20260926 07:34:41.114184 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:34:41.115012 23413 external_mini_cluster.cc:1428] Setting key dc242933a8b2131d4fafcf05599215d0
I20260926 07:34:41.115626 1299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:41.128849 1299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59: Bootstrap starting.
Sep 26 07:34:41 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:41.132700 1299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:41.133797 1299 log.cc:824] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:41.136790 1299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59: No bootstrap required, opened a new log
I20260926 07:34:41.137629 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.117163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.22.221.126:37931)
0926 07:34:41.117948 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.117958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.117997 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:34:41.120252 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.120272 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.120654 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.121050 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:41.122540 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.123868 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.125115 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.125802 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.126011 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.130210 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:41.130217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:41.130232 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:41.130267 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:41.133522 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:41.134265 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:41.134283 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:41.134289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:41.134453 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:41.134838 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:41.134847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:41.134855 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:41.135420 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:41.135722 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.136164 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.136595 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":389,"threads_started":1}
I20260926 07:34:41.144486 1299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3916ff84d3f84c99bf24d30463d7af59" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37931 } }
I20260926 07:34:41.144879 1299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:41.145016 1299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3916ff84d3f84c99bf24d30463d7af59, State: Initialized, Role: FOLLOWER
I20260926 07:34:41.145550 1299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3916ff84d3f84c99bf24d30463d7af59" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37931 } }
I20260926 07:34:41.145826 1299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:41.145975 1299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:41.146108 1299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:41.148309 1299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3916ff84d3f84c99bf24d30463d7af59" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37931 } }
I20260926 07:34:41.148839 1299 leader_election.cc:302] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3916ff84d3f84c99bf24d30463d7af59; no voters:
I20260926 07:34:41.149413 1299 leader_election.cc:288] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:41.149554 1304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:41.150314 1304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [term 1 LEADER]: Becoming Leader. State: Replica: 3916ff84d3f84c99bf24d30463d7af59, State: Running, Role: LEADER
I20260926 07:34:41.150866 1304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3916ff84d3f84c99bf24d30463d7af59" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37931 } }
I20260926 07:34:41.151464 1299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:41.153452 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3916ff84d3f84c99bf24d30463d7af59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3916ff84d3f84c99bf24d30463d7af59" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37931 } } }
I20260926 07:34:41.153796 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:41.154281 1306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3916ff84d3f84c99bf24d30463d7af59. Latest consensus state: current_term: 1 leader_uuid: "3916ff84d3f84c99bf24d30463d7af59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3916ff84d3f84c99bf24d30463d7af59" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37931 } } }
I20260926 07:34:41.154677 1306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:41.154919 1311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:41.165884 1311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:41.180402 1311 catalog_manager.cc:1347] Generated new cluster ID: 2a3a00ea66b14deda0de6bc1214152c6
I20260926 07:34:41.180569 1311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:41.200533 1311 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:41.201802 1311 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:41.214483 1311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59: Generated new TSK 0
I20260926 07:34:41.215539 1311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:34:41.279111 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37931
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:41.601491 1326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:41.601814 1326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:41.601982 1326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:41.609090 1326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:41.609229 1326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:41.609407 1326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:34:41.618769 1326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:41.620013 1326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:41.621294 1326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:41.634482 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:41.634455 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:41.634382 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:41.635113 1326 server_base.cc:1034] running on GCE node
I20260926 07:34:41.635776 1326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:41.636699 1326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:41.637919 1326 hybrid_clock.cc:648] HybridClock initialized: now 1790408081637861 us; error 56 us; skew 500 ppm
I20260926 07:34:41.638226 1326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:41 dist-test-slave-qlq8 krb5kdc[1219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408081, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:41.641561 1326 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:34:41.642953 1326 webserver.cc:460] Webserver started at http://127.22.221.65:39867/ using document root <none> and password file <none>
I20260926 07:34:41.643677 1326 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:41.643815 1326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:41.644105 1326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:41.646839 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "a9d3a3cc5d7941b090016a23c9cadb00"
format_stamp: "Formatted at 2026-09-26 07:34:41 on dist-test-slave-qlq8"
server_key: "867f6a42ecd5827d421366569782c3df"
server_key_iv: "9b8ee527b002b3a2878ebea0158546ed"
server_key_version: "encryptionkey@0"
I20260926 07:34:41.647624 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "a9d3a3cc5d7941b090016a23c9cadb00"
format_stamp: "Formatted at 2026-09-26 07:34:41 on dist-test-slave-qlq8"
server_key: "867f6a42ecd5827d421366569782c3df"
server_key_iv: "9b8ee527b002b3a2878ebea0158546ed"
server_key_version: "encryptionkey@0"
I20260926 07:34:41.652647 1326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:34:41.655930 1341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:41.657225 1326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:34:41.657392 1326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "a9d3a3cc5d7941b090016a23c9cadb00"
format_stamp: "Formatted at 2026-09-26 07:34:41 on dist-test-slave-qlq8"
server_key: "867f6a42ecd5827d421366569782c3df"
server_key_iv: "9b8ee527b002b3a2878ebea0158546ed"
server_key_version: "encryptionkey@0"
I20260926 07:34:41.657574 1326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:41.669276 1326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:41.673367 1326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:41.673666 1326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:41.674562 1326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:41.675905 1326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:41.676003 1326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:41.676141 1326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:41.676209 1326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:41.708632 1326 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:34859
I20260926 07:34:41.708671 1453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:34859 every 8 connection(s)
I20260926 07:34:41.709870 1326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:41.711513 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 1326
I20260926 07:34:41.711748 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:41.712586 23413 external_mini_cluster.cc:1428] Setting key ac554068c6ffa85768394c7cbda8e9f5
Sep 26 07:34:41 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408081, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:41.732402 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.712555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:56237 (local address 127.22.221.126:37931)
0926 07:34:41.713041 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.713046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.716172 (+ 3126us) server_negotiation.cc:407] Connection header received
0926 07:34:41.716224 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.716228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.716301 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.716384 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:41.718426 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.719113 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.721612 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.721992 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.722086 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.725557 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:41.725565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:41.725573 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:41.725617 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:41.729095 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:41.729802 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:41.729810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:41.729816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:41.729965 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:41.730375 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:41.730383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:41.730389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:41.730753 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:41.730903 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.731646 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.731913 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":267,"threads_started":1}
I20260926 07:34:41.732964 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.713237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37931 (local address 127.22.221.65:56237)
0926 07:34:41.714042 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:41.714097 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:41.715896 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:41.716677 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:41.716692 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:41.717458 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:41.718244 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:41.718274 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.719298 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.719306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:41.721458 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:41.721482 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.722210 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.722219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:41.722535 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.723215 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:41.723281 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:41.725339 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:41.729238 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:41.729243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:41.729255 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:41.729667 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:41.730070 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:41.730073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:41.730075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:41.730250 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:41.730887 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:41.730893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:41.731203 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:34:41.731484 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 07:34:41.731864 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":333,"threads_started":1}
I20260926 07:34:41.734522 1454 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37931
I20260926 07:34:41.734890 1454 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:41.735888 1454 heartbeater.cc:511] Master 127.22.221.126:37931 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:34:41.738093 1264 ts_manager.cc:194] Registered new tserver with Master: a9d3a3cc5d7941b090016a23c9cadb00 (127.22.221.65:34859)
I20260926 07:34:41.739733 1264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:56237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:34:41.767012 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37931
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:42.095714 1460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:42.095988 1460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:42.096140 1460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:42.103384 1460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:42.103518 1460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:42.103711 1460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:34:42.113535 1460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:42.114777 1460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:42.116137 1460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:42.129170 1468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:42.129247 1465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:42.129283 1466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:42.130106 1460 server_base.cc:1034] running on GCE node
I20260926 07:34:42.130693 1460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:42.131651 1460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:42.132869 1460 hybrid_clock.cc:648] HybridClock initialized: now 1790408082132830 us; error 37 us; skew 500 ppm
I20260926 07:34:42.133168 1460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:42 dist-test-slave-qlq8 krb5kdc[1219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:42.136217 1460 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:34:42.137859 1460 webserver.cc:460] Webserver started at http://127.22.221.66:38939/ using document root <none> and password file <none>
I20260926 07:34:42.138552 1460 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:42.138700 1460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:42.138988 1460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:42.141953 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "fedc6dc9199847bc944f27740ce29bcf"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-qlq8"
server_key: "42b997c7a93ba8d5a583c69e37fdc831"
server_key_iv: "82b20ebea86a335d2664d98f2a6c1013"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.142911 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "fedc6dc9199847bc944f27740ce29bcf"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-qlq8"
server_key: "42b997c7a93ba8d5a583c69e37fdc831"
server_key_iv: "82b20ebea86a335d2664d98f2a6c1013"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.148530 1460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:34:42.151952 1475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.153326 1460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:34:42.153493 1460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "fedc6dc9199847bc944f27740ce29bcf"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-qlq8"
server_key: "42b997c7a93ba8d5a583c69e37fdc831"
server_key_iv: "82b20ebea86a335d2664d98f2a6c1013"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.153687 1460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:42.170881 1460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:42.175207 1460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:42.175530 1460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:42.176411 1460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:42.177707 1460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:42.177786 1460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.177887 1460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:42.177934 1460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.210408 1460 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:33031
I20260926 07:34:42.210448 1587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:33031 every 8 connection(s)
I20260926 07:34:42.211723 1460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:42.219580 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 1460
I20260926 07:34:42.219776 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:42.220189 23413 external_mini_cluster.cc:1428] Setting key 6893bded831182ff8fa9ecb41dd7e21b
Sep 26 07:34:42 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:42.232571 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.214255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:44225 (local address 127.22.221.126:37931)
0926 07:34:42.214430 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.214435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.215818 (+ 1383us) server_negotiation.cc:407] Connection header received
0926 07:34:42.217596 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.217602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.217691 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.217802 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:42.219278 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.220070 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.221944 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.222432 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.222536 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.226666 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:42.226671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:42.226674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:42.226707 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:42.229349 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.230294 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.230297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.230300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.230419 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.230901 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.230904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.230907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.231161 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:42.231344 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.232128 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.232310 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:42.234097 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.214945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37931 (local address 127.22.221.66:44225)
0926 07:34:42.215662 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.215702 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.217333 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.218046 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.218054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.218522 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:42.219071 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.219088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.220221 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.220232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.221779 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.221806 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.222669 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.222679 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.222993 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.223760 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:42.223798 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:42.226428 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:42.229502 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:42.229513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:42.229532 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:42.230170 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:42.230531 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:42.230539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:42.230545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:42.230796 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:42.231301 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:42.231312 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:42.231757 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.232075 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.232645 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":322,"threads_started":1}
I20260926 07:34:42.235560 1588 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37931
I20260926 07:34:42.235880 1588 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:42.236752 1588 heartbeater.cc:511] Master 127.22.221.126:37931 requested a full tablet report, sending...
I20260926 07:34:42.238283 1264 ts_manager.cc:194] Registered new tserver with Master: fedc6dc9199847bc944f27740ce29bcf (127.22.221.66:33031)
I20260926 07:34:42.239080 1264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:44225
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:34:42.273995 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37931
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:42.596901 1593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:42.597208 1593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:42.597365 1593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:42.604646 1593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:42.604786 1593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:42.604969 1593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:34:42.614473 1593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:34:42.615763 1593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:42.617024 1593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:42.629880 1598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:42.629930 1599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:42.629896 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:42.631386 1593 server_base.cc:1034] running on GCE node
I20260926 07:34:42.631920 1593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:42.632737 1593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:42.633980 1593 hybrid_clock.cc:648] HybridClock initialized: now 1790408082633920 us; error 50 us; skew 500 ppm
I20260926 07:34:42.634310 1593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:34:42 dist-test-slave-qlq8 krb5kdc[1219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:42.637336 1593 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:34:42.638582 1593 webserver.cc:460] Webserver started at http://127.22.221.67:46559/ using document root <none> and password file <none>
I20260926 07:34:42.639204 1593 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:42.639367 1593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:42.639614 1593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:42.642426 1593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "f221b7bcaa51473e8c894b69cdf3a2d9"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-qlq8"
server_key: "291fb69d7d309739f7e59e96a0ac2ec4"
server_key_iv: "656fa21f75b81bcf89fb327871d1ea0a"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.643123 1593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "f221b7bcaa51473e8c894b69cdf3a2d9"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-qlq8"
server_key: "291fb69d7d309739f7e59e96a0ac2ec4"
server_key_iv: "656fa21f75b81bcf89fb327871d1ea0a"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.648200 1593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 07:34:42.651402 1608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.652803 1593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:34:42.652968 1593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "f221b7bcaa51473e8c894b69cdf3a2d9"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-qlq8"
server_key: "291fb69d7d309739f7e59e96a0ac2ec4"
server_key_iv: "656fa21f75b81bcf89fb327871d1ea0a"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.653155 1593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:42.664618 1593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:42.668687 1593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:42.668989 1593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:42.669870 1593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:42.671097 1593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:42.671195 1593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.671368 1593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:42.671440 1593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.703090 1593 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:33773
I20260926 07:34:42.703132 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:33773 every 8 connection(s)
I20260926 07:34:42.704344 1593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:42.704612 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 1593
I20260926 07:34:42.704804 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:42.705371 23413 external_mini_cluster.cc:1428] Setting key 03359cb7571abd13ddcfb4bc8a8604ee
Sep 26 07:34:42 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:34:42.724098 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.707289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:57121 (local address 127.22.221.126:37931)
0926 07:34:42.707503 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.707509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.708849 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 07:34:42.710620 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.710626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.710701 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.710806 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:42.712245 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.713358 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.714772 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.715139 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.715291 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.718376 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:42.718381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:42.718384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:42.718416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:42.721137 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.721837 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.721847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.721849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.721969 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.722377 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.722382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.722385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.722645 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:42.722802 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.723511 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.723732 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:34:42.725009 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.707772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37931 (local address 127.22.221.67:57121)
0926 07:34:42.708680 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.708723 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.710369 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.710994 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.711001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.711545 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:42.712085 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.712103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.713480 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.713484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.714652 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.714668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.715354 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.715359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.715542 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.716137 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:42.716157 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:42.718171 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:42.721279 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:42.721289 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:42.721303 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:42.721703 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:42.722075 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:42.722082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:42.722085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:42.722259 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:42.722784 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:42.722791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:42.723082 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.723374 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.723867 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":466,"threads_started":1}
I20260926 07:34:42.726228 1721 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37931
I20260926 07:34:42.726516 1721 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:42.727180 1721 heartbeater.cc:511] Master 127.22.221.126:37931 requested a full tablet report, sending...
I20260926 07:34:42.728494 1264 ts_manager.cc:194] Registered new tserver with Master: f221b7bcaa51473e8c894b69cdf3a2d9 (127.22.221.67:33773)
I20260926 07:34:42.729310 1264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:57121
I20260926 07:34:42.732887 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:42.742661 1454 heartbeater.cc:503] Master 127.22.221.126:37931 was elected leader, sending a full tablet report...
I20260926 07:34:42.748906 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.737140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.22.221.126:37931)
0926 07:34:42.737298 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.737302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.737756 (+ 454us) server_negotiation.cc:407] Connection header received
0926 07:34:42.737867 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.737870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.737925 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.738001 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:42.739357 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.740043 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.741222 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.741602 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.741757 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.744161 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:42.744167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:42.744172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:42.744208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:42.746675 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.747196 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.747199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.747202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.747400 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.747705 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.747708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.747710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.747979 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:42.748163 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.748416 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.748569 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:42.752781 1264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:42.759873 1264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:42.784291 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.771122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:33773 (local address 127.0.0.1:42208)
0926 07:34:42.771862 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.771899 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.772083 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.773660 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.773671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.773711 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:42.774314 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.774331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.777319 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.777328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.780493 (+ 3165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.780515 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.783076 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.783085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.783418 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.783512 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.783654 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.783899 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":874,"threads_started":1}
I20260926 07:34:42.785804 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.771766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.22.221.67:33773)
0926 07:34:42.772923 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.772933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.772966 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:34:42.773059 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.773070 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.773374 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.774568 (+ 1194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:42.774907 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.777125 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.780798 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.782780 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.784719 (+ 1939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.785017 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.785105 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.785349 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":1130,"threads_started":1}
I20260926 07:34:42.789881 1656 tablet_service.cc:1467] Processing CreateTablet for tablet 43921ae49ed44baaa5581307615dc970 (DEFAULT_TABLE table=test-table [id=3cd332f1641c4ccfaacc013b7eac1b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:42.791462 1656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43921ae49ed44baaa5581307615dc970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:42.792779 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.771217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:33031 (local address 127.0.0.1:39264)
0926 07:34:42.783684 (+ 12467us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.783710 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.783868 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.786335 (+ 2467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.786338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.786358 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:42.786596 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.786603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.788447 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.788455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.790163 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.790174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.791988 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.791994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.792149 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.792183 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.792295 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.792425 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12285,"thread_start_us":12152,"threads_started":1}
I20260926 07:34:42.792840 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.771194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.22.221.66:33031)
0926 07:34:42.785566 (+ 14372us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.785574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.785607 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:34:42.785692 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.785697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.785933 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.786136 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:42.786753 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.788315 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.790323 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.791787 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.791924 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.792102 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.792347 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.792512 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14228,"thread_start_us":14081,"threads_started":1}
I20260926 07:34:42.794591 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.772777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:34859 (local address 127.0.0.1:58398)
0926 07:34:42.784089 (+ 11312us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.784106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.784208 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.787926 (+ 3718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.787931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.787956 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:42.788294 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.788304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.790163 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.790168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.791650 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.791662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.793819 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.793826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.793994 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.794029 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.794121 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.794246 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11172,"thread_start_us":11524,"threads_started":1}
I20260926 07:34:42.794706 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.773450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.22.221.65:34859)
0926 07:34:42.787354 (+ 13904us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.787362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.787389 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:34:42.787477 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.787484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.787702 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.787903 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:42.788510 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.790012 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.791802 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.793607 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.793746 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.793985 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.794187 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.794365 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13694,"thread_start_us":14592,"threads_started":1}
I20260926 07:34:42.796926 1523 tablet_service.cc:1467] Processing CreateTablet for tablet 43921ae49ed44baaa5581307615dc970 (DEFAULT_TABLE table=test-table [id=3cd332f1641c4ccfaacc013b7eac1b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:42.798520 1523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43921ae49ed44baaa5581307615dc970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:42.799321 1389 tablet_service.cc:1467] Processing CreateTablet for tablet 43921ae49ed44baaa5581307615dc970 (DEFAULT_TABLE table=test-table [id=3cd332f1641c4ccfaacc013b7eac1b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:42.800853 1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43921ae49ed44baaa5581307615dc970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:42.804137 1739 tablet_bootstrap.cc:492] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Bootstrap starting.
I20260926 07:34:42.807229 1739 tablet_bootstrap.cc:654] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:42.808281 1739 log.cc:824] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:42.810930 1739 tablet_bootstrap.cc:492] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: No bootstrap required, opened a new log
I20260926 07:34:42.811576 1739 ts_tablet_manager.cc:1397] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:34:42.814407 1741 tablet_bootstrap.cc:492] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Bootstrap starting.
I20260926 07:34:42.816457 1742 tablet_bootstrap.cc:492] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Bootstrap starting.
I20260926 07:34:42.818171 1741 tablet_bootstrap.cc:654] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:42.819394 1742 tablet_bootstrap.cc:654] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:42.819406 1741 log.cc:824] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:42.820609 1742 log.cc:824] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:42.821421 1739 raft_consensus.cc:348] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.821871 1739 raft_consensus.cc:374] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:42.822033 1739 raft_consensus.cc:729] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f221b7bcaa51473e8c894b69cdf3a2d9, State: Initialized, Role: FOLLOWER
I20260926 07:34:42.822772 1739 consensus_queue.cc:260] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.823220 1741 tablet_bootstrap.cc:492] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: No bootstrap required, opened a new log
I20260926 07:34:42.823576 1741 ts_tablet_manager.cc:1397] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 07:34:42.823791 1742 tablet_bootstrap.cc:492] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: No bootstrap required, opened a new log
I20260926 07:34:42.824069 1742 ts_tablet_manager.cc:1397] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260926 07:34:42.824152 1721 heartbeater.cc:503] Master 127.22.221.126:37931 was elected leader, sending a full tablet report...
I20260926 07:34:42.825419 1739 ts_tablet_manager.cc:1428] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260926 07:34:42.832531 1742 raft_consensus.cc:348] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.832913 1742 raft_consensus.cc:374] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:42.833016 1742 raft_consensus.cc:729] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d3a3cc5d7941b090016a23c9cadb00, State: Initialized, Role: FOLLOWER
I20260926 07:34:42.833649 1742 consensus_queue.cc:260] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.833591 1741 raft_consensus.cc:348] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.833995 1741 raft_consensus.cc:374] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:42.834096 1741 raft_consensus.cc:729] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedc6dc9199847bc944f27740ce29bcf, State: Initialized, Role: FOLLOWER
I20260926 07:34:42.834895 1741 consensus_queue.cc:260] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.835672 1742 ts_tablet_manager.cc:1428] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 07:34:42.836915 1588 heartbeater.cc:503] Master 127.22.221.126:37931 was elected leader, sending a full tablet report...
I20260926 07:34:42.837363 1741 ts_tablet_manager.cc:1428] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
W20260926 07:34:42.956132 1722 tablet.cc:2369] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:42.956172 1745 raft_consensus.cc:482] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:42.956501 1745 raft_consensus.cc:504] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.958333 1745 leader_election.cc:288] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fedc6dc9199847bc944f27740ce29bcf (127.22.221.66:33031), a9d3a3cc5d7941b090016a23c9cadb00 (127.22.221.65:34859)
W20260926 07:34:42.963024 1455 tablet.cc:2369] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:34:42.963814 1589 tablet.cc:2369] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:42.965070 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.958588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:34859 (local address 127.22.221.67:53553)
0926 07:34:42.958851 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.958872 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.959009 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.959384 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.959389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.959421 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:42.959616 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.959627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.960604 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.960610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.962545 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.962563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.964354 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.964360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.964542 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.964567 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.964671 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.964802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 07:34:42.965067 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.958817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:53553 (local address 127.22.221.65:34859)
0926 07:34:42.959005 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.959011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.959032 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:34:42.959152 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.959155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.959212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.959363 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:42.959761 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.960473 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.962693 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.964191 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.964366 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.964440 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.964697 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.964829 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:34:42.965850 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.958847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:58603 (local address 127.22.221.66:33031)
0926 07:34:42.959046 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.959054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.959742 (+ 688us) server_negotiation.cc:407] Connection header received
0926 07:34:42.959942 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.959947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.960013 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.960108 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:42.960663 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.961435 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.963814 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.964950 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.965115 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.965189 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.965478 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.965606 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:34:42.965950 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.958588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:33031 (local address 127.22.221.67:58603)
0926 07:34:42.959638 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.959652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.959811 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.960120 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.960124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.960146 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:42.960521 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.960530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.961587 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.961593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.963609 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.963624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.965160 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.965167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.965323 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.965344 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.965441 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.965560 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":355,"thread_start_us":360,"threads_started":1}
I20260926 07:34:42.966352 1409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43921ae49ed44baaa5581307615dc970" candidate_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" is_pre_election: true
I20260926 07:34:42.966849 1409 raft_consensus.cc:2454] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f221b7bcaa51473e8c894b69cdf3a2d9 in term 0.
I20260926 07:34:42.966857 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43921ae49ed44baaa5581307615dc970" candidate_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fedc6dc9199847bc944f27740ce29bcf" is_pre_election: true
I20260926 07:34:42.967365 1543 raft_consensus.cc:2454] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f221b7bcaa51473e8c894b69cdf3a2d9 in term 0.
I20260926 07:34:42.967574 1609 leader_election.cc:302] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9d3a3cc5d7941b090016a23c9cadb00, f221b7bcaa51473e8c894b69cdf3a2d9; no voters:
I20260926 07:34:42.967967 1745 raft_consensus.cc:2781] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:42.968122 1745 raft_consensus.cc:482] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:42.968232 1745 raft_consensus.cc:3037] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:42.970403 1745 raft_consensus.cc:504] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.971014 1745 leader_election.cc:288] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [CANDIDATE]: Term 1 election: Requested vote from peers fedc6dc9199847bc944f27740ce29bcf (127.22.221.66:33031), a9d3a3cc5d7941b090016a23c9cadb00 (127.22.221.65:34859)
I20260926 07:34:42.971437 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43921ae49ed44baaa5581307615dc970" candidate_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fedc6dc9199847bc944f27740ce29bcf"
I20260926 07:34:42.971539 1409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43921ae49ed44baaa5581307615dc970" candidate_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d3a3cc5d7941b090016a23c9cadb00"
I20260926 07:34:42.971649 1543 raft_consensus.cc:3037] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:42.971740 1409 raft_consensus.cc:3037] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:42.974900 1543 raft_consensus.cc:2454] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f221b7bcaa51473e8c894b69cdf3a2d9 in term 1.
I20260926 07:34:42.974895 1409 raft_consensus.cc:2454] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f221b7bcaa51473e8c894b69cdf3a2d9 in term 1.
I20260926 07:34:42.975310 1609 leader_election.cc:302] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9d3a3cc5d7941b090016a23c9cadb00, f221b7bcaa51473e8c894b69cdf3a2d9; no voters:
I20260926 07:34:42.975586 1745 raft_consensus.cc:2781] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:42.976038 1745 raft_consensus.cc:686] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 1 LEADER]: Becoming Leader. State: Replica: f221b7bcaa51473e8c894b69cdf3a2d9, State: Running, Role: LEADER
I20260926 07:34:42.976480 1745 consensus_queue.cc:237] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } }
I20260926 07:34:42.980907 1263 catalog_manager.cc:5495] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f221b7bcaa51473e8c894b69cdf3a2d9 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f221b7bcaa51473e8c894b69cdf3a2d9" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:43.021121 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.015755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.22.221.67:33773)
0926 07:34:43.015986 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.015993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.016090 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:34:43.016328 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.016332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.016406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.016558 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:43.016942 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.017668 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.018992 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.019414 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.019547 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.020049 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:43.020540 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:43.020698 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.020779 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.020859 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:34:43.029075 1409 raft_consensus.cc:1261] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 1 FOLLOWER]: Refusing update from remote peer f221b7bcaa51473e8c894b69cdf3a2d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:43.029075 1543 raft_consensus.cc:1261] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 1 FOLLOWER]: Refusing update from remote peer f221b7bcaa51473e8c894b69cdf3a2d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:43.030016 1749 consensus_queue.cc:1035] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fedc6dc9199847bc944f27740ce29bcf" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:43.030400 1745 consensus_queue.cc:1035] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9d3a3cc5d7941b090016a23c9cadb00" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:43.041445 1752 mvcc.cc:205] Tried to move back new op lower bound from 7333511508077506560 to 7333511507874996224. Current Snapshot: MvccSnapshot[applied={T|T < 7333511508077506560}]
I20260926 07:34:43.042879 1754 mvcc.cc:205] Tried to move back new op lower bound from 7333511508077506560 to 7333511507874996224. Current Snapshot: MvccSnapshot[applied={T|T < 7333511508077506560}]
I20260926 07:34:43.056339 1263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:43.056654 1263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:43.060425 1263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3916ff84d3f84c99bf24d30463d7af59: Sending DeleteTablet for 3 replicas of tablet 43921ae49ed44baaa5581307615dc970
I20260926 07:34:43.061662 1523 tablet_service.cc:1514] Processing DeleteTablet for tablet 43921ae49ed44baaa5581307615dc970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:43 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:39264
I20260926 07:34:43.061725 1656 tablet_service.cc:1514] Processing DeleteTablet for tablet 43921ae49ed44baaa5581307615dc970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:43 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:42208
I20260926 07:34:43.062726 1389 tablet_service.cc:1514] Processing DeleteTablet for tablet 43921ae49ed44baaa5581307615dc970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:43 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:58398
I20260926 07:34:43.063619 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 1326
I20260926 07:34:43.078348 1763 tablet_replica.cc:330] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: stopping tablet replica
I20260926 07:34:43.078848 1763 raft_consensus.cc:2229] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:43.079505 1763 raft_consensus.cc:2258] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:43.081934 1763 ts_tablet_manager.cc:1905] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:43.083346 1761 tablet_replica.cc:330] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: stopping tablet replica
I20260926 07:34:43.083938 1762 tablet_replica.cc:330] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: stopping tablet replica
I20260926 07:34:43.084540 1762 raft_consensus.cc:2229] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:43.085223 1762 raft_consensus.cc:2258] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:43.085906 1761 raft_consensus.cc:2229] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:43.086438 1761 raft_consensus.cc:2258] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:43.087702 1762 ts_tablet_manager.cc:1905] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:43.088497 1761 ts_tablet_manager.cc:1905] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:43.089028 1763 ts_tablet_manager.cc:1918] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:43.089188 1763 log.cc:1196] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/43921ae49ed44baaa5581307615dc970
I20260926 07:34:43.089715 1763 ts_tablet_manager.cc:1939] T 43921ae49ed44baaa5581307615dc970 P a9d3a3cc5d7941b090016a23c9cadb00: Deleting consensus metadata
I20260926 07:34:43.091604 1248 catalog_manager.cc:4841] TS a9d3a3cc5d7941b090016a23c9cadb00 (127.22.221.65:34859): tablet 43921ae49ed44baaa5581307615dc970 (table test-table [id=3cd332f1641c4ccfaacc013b7eac1b08]) successfully deleted
I20260926 07:34:43.095383 1761 ts_tablet_manager.cc:1918] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:43.095551 1761 log.cc:1196] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/43921ae49ed44baaa5581307615dc970
Sep 26 07:34:43 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:34:43.096086 1761 ts_tablet_manager.cc:1939] T 43921ae49ed44baaa5581307615dc970 P fedc6dc9199847bc944f27740ce29bcf: Deleting consensus metadata
I20260926 07:34:43.097383 1762 ts_tablet_manager.cc:1918] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:43.097868 1249 catalog_manager.cc:4841] TS fedc6dc9199847bc944f27740ce29bcf (127.22.221.66:33031): tablet 43921ae49ed44baaa5581307615dc970 (table test-table [id=3cd332f1641c4ccfaacc013b7eac1b08]) successfully deleted
I20260926 07:34:43.098184 1762 log.cc:1196] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/43921ae49ed44baaa5581307615dc970
I20260926 07:34:43.098733 1762 ts_tablet_manager.cc:1939] T 43921ae49ed44baaa5581307615dc970 P f221b7bcaa51473e8c894b69cdf3a2d9: Deleting consensus metadata
I20260926 07:34:43.100337 1248 catalog_manager.cc:4841] TS f221b7bcaa51473e8c894b69cdf3a2d9 (127.22.221.67:33773): tablet 43921ae49ed44baaa5581307615dc970 (table test-table [id=3cd332f1641c4ccfaacc013b7eac1b08]) successfully deleted
I20260926 07:34:43.103035 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.064682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.22.221.65:34859)
0926 07:34:43.064916 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.064926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.070939 (+ 6013us) server_negotiation.cc:407] Connection header received
0926 07:34:43.071999 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.072008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.072119 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.072209 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.076567 (+ 4358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.078132 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.082177 (+ 4045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.082826 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.082933 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.096990 (+ 14057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.097007 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.097025 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.097075 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.100234 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.100962 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.100969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.100971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.101099 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.101492 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.101496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.101498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.101839 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.102145 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.102448 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.102670 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:34:43.103794 1449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:43.192757 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 1326
I20260926 07:34:43.215870 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 1460
Sep 26 07:34:43 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:34:43.234494 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.216673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.22.221.66:33031)
0926 07:34:43.216857 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.216866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.216963 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:34:43.217135 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.217141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.217226 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.217344 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.218877 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.220183 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.221968 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.222599 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.222733 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.227911 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.227923 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.227940 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.228001 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.231753 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.232615 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.232622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.232624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.232746 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.233129 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.233132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.233134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.233499 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.233711 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.234037 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.234208 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:43.235183 1583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:43.324072 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 1460
I20260926 07:34:43.342597 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 1593
Sep 26 07:34:43 dist-test-slave-qlq8 krb5kdc[1219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:34:43.360706 1736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.343554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.22.221.67:33773)
0926 07:34:43.343709 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.343713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.343842 (+ 129us) server_negotiation.cc:407] Connection header received
0926 07:34:43.344007 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.344013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.344129 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.344258 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.345497 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.346710 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.348033 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.348758 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.348894 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.353611 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.353625 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.353643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.353697 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.357738 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.358369 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.358382 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.358390 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.358582 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.358965 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.358970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.358973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.359529 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.359827 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.360046 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.360288 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:34:43.361557 1716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:43.480839 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 1593
I20260926 07:34:43.499606 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 1232
I20260926 07:34:43.500695 1294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:34:43.609705 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 1232
2026-09-26T07:34:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4271 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:43 dist-test-slave-qlq8 krb5kdc[1771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:43 dist-test-slave-qlq8 krb5kdc[1771](info): set up 2 sockets
Sep 26 07:34:43 dist-test-slave-qlq8 krb5kdc[1771](info): commencing operation
krb5kdc: starting...
W20260926 07:34:44.772420 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:34:44 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/rangeradmin_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/rangeradmin_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/rangerlookup_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/rangerlookup_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
I20260926 07:34:45.080175 23413 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdo7xD1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/postgres -l logfile start
2026-09-26 07:34:47.627 UTC [1801] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:34:47.627 UTC [1801] LOG: listening on IPv4 address "127.22.221.84", port 36797
2026-09-26 07:34:47.631 UTC [1801] LOG: listening on Unix socket "/tmp/.s.PGSQL.36797"
2026-09-26 07:34:47.653 UTC [1804] LOG: database system was shut down at 2026-09-26 07:34:46 UTC
2026-09-26 07:34:47.656 UTC [1801] LOG: database system is ready to accept connections
I20260926 07:34:48.632105 23413 mini_postgres.cc:96] Postgres bound to 36797
2026-09-26 07:34:48.649 UTC [1812] FATAL: database "slave" does not exist
127.22.221.84:36797 - accepting connections
I20260926 07:34:48.650561 23413 mini_ranger.cc:161] Starting Ranger...
I20260926 07:34:48.676762 23413 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:34:49.176314 23413 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:34:49.176450 23413 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-admin
2026-09-26 07:34:49,472 [I] DB FLAVOR :POSTGRES
2026-09-26 07:34:49,473 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:34:49,473 [I] Checking connection..
2026-09-26 07:34:49,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:34:49,712 [I] Checking connection passed.
2026-09-26 07:34:49,713 [I] --------- Verifying version history table ---------
2026-09-26 07:34:49,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:34:49,975 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:34:49,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:34:50,210 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:34:50,210 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:34:50,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:34:50.448 UTC [1876] WARNING: there is no transaction in progress
2026-09-26 07:34:50,454 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:34:50,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:34:50,700 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:34:50,700 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:34:50,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:34:50,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:34:51,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qlq8.c.gcp-upstream.internal','N') ;"
2026-09-26 07:34:51,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:34:51,644 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:34:51,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:34:51,890 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:34:51,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:34:52,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:34:52,370 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:34:52,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:34:53.344 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.361 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.379 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.395 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.412 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.430 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.494 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.501 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.516 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.527 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.539 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.554 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.564 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.570 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.578 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:53.585 UTC [1984] WARNING: there is no transaction in progress
2026-09-26 07:34:54,226 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:34:54,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:34:54,483 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:34:54,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:34:54,732 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:34:54,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:34:54,988 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:34:54,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:34:55,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:34:55,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qlq8.c.gcp-upstream.internal';"
2026-09-26 07:34:55,729 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:34:55,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:34:55,983 [I] DB_PATCHES have already been applied
I20260926 07:34:55.988960 23413 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdo7xD1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:34:55.989077 23413 mini_ranger.cc:193] Using host: 127.22.221.84
I20260926 07:34:56.004375 23413 mini_ranger.cc:227] Ranger admin URL: http://127.22.221.84:37525
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:34:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:34:57.147205 23413 mini_ranger.cc:160] Time spent starting Ranger: real 8.497s user 0.001s sys 0.039s
Sep 26, 2026 7:35:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:35:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 7:35:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:35:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:35:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:35:18.045542 23413 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/rangerkms_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/rangerkms_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.221.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.221.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:35:18 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408118, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:35:18.212713 23413 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:35:18.234256 23413 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:35:19.134366 23413 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:35:19.134526 23413 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms
I20260926 07:35:19.134671 23413 mini_ranger_kms.cc:227] Using postgres at 127.22.221.84:36797
2026-09-26 07:35:19,559 [I] DB FLAVOR :POSTGRES
2026-09-26 07:35:19,559 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:35:19,559 [I] Checking connection
2026-09-26 07:35:19,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:35:19,811 [I] connection success
2026-09-26 07:35:19,812 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:35:19,812 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:35:19,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:35:20,053 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:35:20,053 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:35:20,053 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:35:20,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:36797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:35:20,301 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:35:20.766376 23413 mini_ranger_kms.cc:310] Created kms service
I20260926 07:35:20.950955 23413 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:35:20.997177 23413 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:35:20 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408118, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:35:21.396987 23413 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:35:21.397323 23413 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:35:21.397377 23413 mini_ranger_kms.cc:240] Using host: 127.22.221.84
I20260926 07:35:21.418527 23413 mini_ranger_kms.cc:276] Ranger KMS PID: 2164
I20260926 07:35:21.418731 23413 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.221.84:38151
Sep 26, 2026 7:35:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:35:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:35:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:35:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:35:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:35:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 07:35:23 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790408123, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.221.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:35:23.707640 23413 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.495s user 0.140s sys 0.306s
I20260926 07:35:23.707929 23413 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:35:23.708141 23413 mini_ranger_kms.cc:401] 127.22.221.84:38151/kms/v1/keys
Sep 26 07:35:24 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408123, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.221.84@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
Sep 26 07:35:24 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408123, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.221.84@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
Sep 26 07:35:26 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:35:26.808035 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:33197
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:33197
--ranger_config_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:38151/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:35:27.138981 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:35:27.139362 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:35:27.139523 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:35:27.146852 2215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:35:27.146992 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:35:27.147089 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:35:27.147183 2215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:35:27.147289 2215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:35:27.157516 2215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.221.84:38151/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:33197
--ranger_config_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:35:27.158818 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:35:27.160167 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:35:27.172461 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:27.172691 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:27.172833 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:35:27.173498 2215 server_base.cc:1034] running on GCE node
I20260926 07:35:27.174292 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:35:27.175758 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:35:27.176985 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790408127176937 us; error 42 us; skew 500 ppm
I20260926 07:35:27.177318 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:35:27 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408127, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:35:27.186025 2215 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:35:27.187613 2215 webserver.cc:460] Webserver started at http://127.22.221.126:41659/ using document root <none> and password file <none>
I20260926 07:35:27.188256 2215 fs_manager.cc:359] Metadata directory not provided
I20260926 07:35:27.188385 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:35:27.188691 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:35:27 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408127, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:35:27.237746 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "6791bd67a4f54460901b33ccd2db9957"
format_stamp: "Formatted at 2026-09-26 07:35:27 on dist-test-slave-qlq8"
server_key: "388d8fae747ecb908c98bea30a9b6fab"
server_key_iv: "d14e4e81d9baa0476989bd01b20f58c4"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:27.238719 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "6791bd67a4f54460901b33ccd2db9957"
format_stamp: "Formatted at 2026-09-26 07:35:27 on dist-test-slave-qlq8"
server_key: "388d8fae747ecb908c98bea30a9b6fab"
server_key_iv: "d14e4e81d9baa0476989bd01b20f58c4"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:27.244275 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:35:27.261132 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:27.262930 2215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:35:27.263123 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "6791bd67a4f54460901b33ccd2db9957"
format_stamp: "Formatted at 2026-09-26 07:35:27 on dist-test-slave-qlq8"
server_key: "388d8fae747ecb908c98bea30a9b6fab"
server_key_iv: "d14e4e81d9baa0476989bd01b20f58c4"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:27.263412 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:35:27.278555 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:35:27.282424 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:35:27.282733 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:35:27.302289 2215 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:33197
I20260926 07:35:27.302307 2281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:33197 every 8 connection(s)
I20260926 07:35:27.303706 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:35:27.307354 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 2215
I20260926 07:35:27.307570 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:35:27.307840 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:35:27.308398 23413 external_mini_cluster.cc:1428] Setting key 12a7a5845e54e1baa6b2948920b14581
I20260926 07:35:27.324211 2282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957: Bootstrap starting.
I20260926 07:35:27.328220 2282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957: Neither blocks nor log segments found. Creating new log.
I20260926 07:35:27.329412 2282 log.cc:824] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957: Log is configured to *not* fsync() on all Append() calls
I20260926 07:35:27.332427 2282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957: No bootstrap required, opened a new log
Sep 26 07:35:27 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:35:27.341387 2282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6791bd67a4f54460901b33ccd2db9957" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33197 } }
I20260926 07:35:27.341785 2282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:35:27.341928 2282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6791bd67a4f54460901b33ccd2db9957, State: Initialized, Role: FOLLOWER
I20260926 07:35:27.342578 2282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6791bd67a4f54460901b33ccd2db9957" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33197 } }
I20260926 07:35:27.342836 2282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:35:27.342983 2282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:35:27.343165 2282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:35:27.345930 2282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6791bd67a4f54460901b33ccd2db9957" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33197 } }
I20260926 07:35:27.346551 2282 leader_election.cc:302] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6791bd67a4f54460901b33ccd2db9957; no voters:
I20260926 07:35:27.347339 2282 leader_election.cc:288] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:35:27.347461 2287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:35:27.348277 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:27.310452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.22.221.126:33197)
0926 07:35:27.323842 (+ 13390us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:27.323859 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:27.323922 (+ 63us) server_negotiation.cc:407] Connection header received
0926 07:35:27.326361 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:27.326391 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:27.326904 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:27.327376 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:27.328961 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:27.330676 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:27.332723 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:27.333765 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:27.333946 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:27.338853 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:27.338863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:27.338886 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:27.338951 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:27.343036 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:27.343985 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:27.343992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:27.344000 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:27.344196 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:27.344633 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:27.344638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:27.344645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:27.345355 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:27.345640 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:27.346225 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 07:35:27.346639 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13129,"thread_start_us":12900,"threads_started":1}
I20260926 07:35:27.354384 2287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [term 1 LEADER]: Becoming Leader. State: Replica: 6791bd67a4f54460901b33ccd2db9957, State: Running, Role: LEADER
I20260926 07:35:27.355002 2287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6791bd67a4f54460901b33ccd2db9957" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33197 } }
I20260926 07:35:27.356007 2282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:35:27.358238 2288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6791bd67a4f54460901b33ccd2db9957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6791bd67a4f54460901b33ccd2db9957" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33197 } } }
I20260926 07:35:27.362267 2288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [sys.catalog]: This master's current role is: LEADER
I20260926 07:35:27.358474 2289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6791bd67a4f54460901b33ccd2db9957. Latest consensus state: current_term: 1 leader_uuid: "6791bd67a4f54460901b33ccd2db9957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6791bd67a4f54460901b33ccd2db9957" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 33197 } } }
I20260926 07:35:27.363200 2289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957 [sys.catalog]: This master's current role is: LEADER
I20260926 07:35:27.365768 2292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:35:27.378676 2292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:35:27.406283 2282 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:35:27.407836 2292 catalog_manager.cc:1347] Generated new cluster ID: 7483a175c67f4617b699a632f21507c1
I20260926 07:35:27.407964 2292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:35:27.448421 2292 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:35:27.450568 2292 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:35:27.461587 2292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957: Generated new TSK 0
I20260926 07:35:27.462869 2292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:35:28.318974 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:38151/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33197
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:35:28 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790408128, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:35:28.768193 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:35:28.768703 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:35:28.768911 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:35:28.778554 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:35:28.778672 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:35:28.778851 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:35:28.789048 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.221.84:38151/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:35:28.790375 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:35:28.791740 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:35:28.823575 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:28.824729 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:28.832234 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:35:28.833528 2327 server_base.cc:1034] running on GCE node
I20260926 07:35:28.834249 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:35:28.835484 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:35:28.836843 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790408128836767 us; error 71 us; skew 500 ppm
I20260926 07:35:28.837232 2327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:35:28 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408128, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:35:28.841041 2327 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:35:28.847143 2327 webserver.cc:460] Webserver started at http://127.22.221.65:40623/ using document root <none> and password file <none>
I20260926 07:35:28.848173 2327 fs_manager.cc:359] Metadata directory not provided
I20260926 07:35:28.848390 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:35:28.848743 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:35:28 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408128, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:35:28.866852 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "9bd0398ea1d347deab0d0580d15ddb0a"
format_stamp: "Formatted at 2026-09-26 07:35:28 on dist-test-slave-qlq8"
server_key: "bfb00f2a7675d389583c76dc64fde6c8"
server_key_iv: "d68450019760aaa74c3d6f7e0729e5ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:28.868175 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "9bd0398ea1d347deab0d0580d15ddb0a"
format_stamp: "Formatted at 2026-09-26 07:35:28 on dist-test-slave-qlq8"
server_key: "bfb00f2a7675d389583c76dc64fde6c8"
server_key_iv: "d68450019760aaa74c3d6f7e0729e5ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:28.893778 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.004s sys 0.020s
I20260926 07:35:28.931973 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:28.934469 2327 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.004s sys 0.016s
I20260926 07:35:28.934753 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "9bd0398ea1d347deab0d0580d15ddb0a"
format_stamp: "Formatted at 2026-09-26 07:35:28 on dist-test-slave-qlq8"
server_key: "bfb00f2a7675d389583c76dc64fde6c8"
server_key_iv: "d68450019760aaa74c3d6f7e0729e5ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:28.935124 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 07:35:28 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408128, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:35:29.017890 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:35:29.026410 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:35:29.026825 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:35:29.034412 2327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:35:29.042747 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:35:29.042912 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:29.043154 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:35:29.043308 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:29.136492 2327 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:34837
I20260926 07:35:29.136543 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:34837 every 8 connection(s)
I20260926 07:35:29.138139 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:35:29.143148 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 2327
I20260926 07:35:29.143431 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:35:29.144311 23413 external_mini_cluster.cc:1428] Setting key 959a25005c5ff9a372165cf64ed7cce2
Sep 26 07:35:29 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408128, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:35:29.190804 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:29.140960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:57047 (local address 127.22.221.126:33197)
0926 07:35:29.158084 (+ 17124us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:29.158094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:29.169789 (+ 11695us) server_negotiation.cc:407] Connection header received
0926 07:35:29.171859 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:29.171865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:29.171974 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:29.172085 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:29.174375 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.175696 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.177877 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.178336 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.178481 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:29.183185 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:29.183192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:29.183199 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:29.183279 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:29.187015 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:29.188036 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:29.188043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:29.188047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:29.188193 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:29.188822 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:29.188829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:29.188835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:29.189128 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:29.189227 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:29.190172 (+ 945us) server_negotiation.cc:299] Negotiation successful
0926 07:35:29.190410 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16914,"thread_start_us":17150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:35:29.192363 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:29.141706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33197 (local address 127.22.221.65:57047)
0926 07:35:29.169586 (+ 27880us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:29.169642 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:29.171578 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:29.172451 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:29.172467 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:29.173194 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:35:29.174151 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:29.174182 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.175896 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.175905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:29.177697 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:29.177710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.178575 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.178582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:29.178918 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:29.179776 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:35:29.179811 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:35:29.182900 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:35:29.187189 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:35:29.187201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:35:29.187219 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:35:29.187878 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:35:29.188340 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:35:29.188344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:35:29.188348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:35:29.188668 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:35:29.189300 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:35:29.189306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:35:29.189635 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 07:35:29.190078 (+ 443us) client_negotiation.cc:240] Negotiation successful
0926 07:35:29.190619 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27491,"thread_start_us":26156,"threads_started":1}
I20260926 07:35:29.194482 2460 heartbeater.cc:347] Connected to a master server at 127.22.221.126:33197
I20260926 07:35:29.194811 2460 heartbeater.cc:464] Registering TS with master...
I20260926 07:35:29.195649 2460 heartbeater.cc:511] Master 127.22.221.126:33197 requested a full tablet report, sending...
I20260926 07:35:29.198112 2247 ts_manager.cc:194] Registered new tserver with Master: 9bd0398ea1d347deab0d0580d15ddb0a (127.22.221.65:34837)
I20260926 07:35:29.200248 2247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:57047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:35:29.227165 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:38151/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33197
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:35:29 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408128, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
W20260926 07:35:29.695647 2466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:35:29.695906 2466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:35:29.696066 2466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:35:29.703126 2466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:35:29.703220 2466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:35:29.703450 2466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:35:29.712932 2466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.221.84:38151/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:35:29.714147 2466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:35:29.715466 2466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:35:29.729434 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:29.729446 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:29.730000 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:35:29.731129 2466 server_base.cc:1034] running on GCE node
I20260926 07:35:29.731851 2466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:35:29.732952 2466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:35:29.734195 2466 hybrid_clock.cc:648] HybridClock initialized: now 1790408129734108 us; error 70 us; skew 500 ppm
I20260926 07:35:29.734560 2466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:35:29 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408129, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:35:29.738458 2466 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:35:29.740136 2466 webserver.cc:460] Webserver started at http://127.22.221.66:44463/ using document root <none> and password file <none>
I20260926 07:35:29.741034 2466 fs_manager.cc:359] Metadata directory not provided
I20260926 07:35:29.741163 2466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:35:29.741533 2466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:35:29 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408129, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:35:29.760546 2466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "f83cf9ed308142bf9de9a0412e707460"
format_stamp: "Formatted at 2026-09-26 07:35:29 on dist-test-slave-qlq8"
server_key: "ba43bb978b34fafbf8adf197c035ef2d"
server_key_iv: "2cbef8aac5df0ae85ab2061985198104"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:29.761637 2466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "f83cf9ed308142bf9de9a0412e707460"
format_stamp: "Formatted at 2026-09-26 07:35:29 on dist-test-slave-qlq8"
server_key: "ba43bb978b34fafbf8adf197c035ef2d"
server_key_iv: "2cbef8aac5df0ae85ab2061985198104"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:29.769408 2466 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260926 07:35:29.788132 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:29.790345 2466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:35:29.790611 2466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "f83cf9ed308142bf9de9a0412e707460"
format_stamp: "Formatted at 2026-09-26 07:35:29 on dist-test-slave-qlq8"
server_key: "ba43bb978b34fafbf8adf197c035ef2d"
server_key_iv: "2cbef8aac5df0ae85ab2061985198104"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:29.790940 2466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:35:29.820832 2466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:35:29.825956 2466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:35:29.826367 2466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:35:29.827601 2466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:35:29.829372 2466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:35:29.829504 2466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 07:35:29.829658 2466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:35:29.829756 2466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:29.866115 2466 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:42927
I20260926 07:35:29.866169 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:42927 every 8 connection(s)
I20260926 07:35:29.867805 2466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:35:29.869078 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 2466
I20260926 07:35:29.869238 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:35:29.869792 23413 external_mini_cluster.cc:1428] Setting key 906991bda11ed0d1d287dbbdea1fc507
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Sep 26 07:35:29 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408129, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:35:29.918715 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:29.883058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:43309 (local address 127.22.221.126:33197)
0926 07:35:29.896078 (+ 13020us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:29.896087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:29.898171 (+ 2084us) server_negotiation.cc:407] Connection header received
0926 07:35:29.900863 (+ 2692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:29.900870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:29.900975 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:29.901110 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:29.903079 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.903919 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.905567 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.905957 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.906098 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:29.911437 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:29.911444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:29.911449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:29.911493 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:29.914918 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:29.915929 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:29.915934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:29.915938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:29.916089 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:29.916640 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:29.916646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:29.916649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:29.917010 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:29.917153 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:29.918134 (+ 981us) server_negotiation.cc:299] Negotiation successful
0926 07:35:29.918353 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12856,"thread_start_us":13020,"threads_started":1}
I20260926 07:35:29.919723 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:29.883696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33197 (local address 127.22.221.66:43309)
0926 07:35:29.897948 (+ 14252us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:29.898013 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:29.900515 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:29.901392 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:29.901405 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:29.902021 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:35:29.902892 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:29.902917 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.904098 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.904102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:29.905420 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:29.905432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:29.906196 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:29.906203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:29.906430 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:29.907112 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:35:29.907135 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:35:29.911166 (+ 4031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:35:29.915075 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:35:29.915087 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:35:29.915110 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:35:29.915795 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:35:29.916211 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:35:29.916218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:35:29.916224 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:35:29.916506 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:35:29.917162 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:35:29.917173 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:35:29.917644 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 07:35:29.917988 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 07:35:29.918461 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13877,"thread_start_us":15480,"threads_started":1}
I20260926 07:35:29.921408 2603 heartbeater.cc:347] Connected to a master server at 127.22.221.126:33197
I20260926 07:35:29.921736 2603 heartbeater.cc:464] Registering TS with master...
I20260926 07:35:29.922695 2603 heartbeater.cc:511] Master 127.22.221.126:33197 requested a full tablet report, sending...
I20260926 07:35:29.924170 2247 ts_manager.cc:194] Registered new tserver with Master: f83cf9ed308142bf9de9a0412e707460 (127.22.221.66:42927)
I20260926 07:35:29.925050 2247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:43309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:35:29.944805 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:38151/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33197
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:35:30.203480 2460 heartbeater.cc:503] Master 127.22.221.126:33197 was elected leader, sending a full tablet report...
W20260926 07:35:30.295500 2609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:35:30.295806 2609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:35:30.295976 2609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:35:30.303993 2609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:35:30.304107 2609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:35:30.304280 2609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:35:30.316978 2609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.221.84:38151/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:33197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:35:30.318100 2609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:35:30.319342 2609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:35:30.331928 2614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:30.331988 2615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:35:30.332111 2617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:35:30.332741 2609 server_base.cc:1034] running on GCE node
I20260926 07:35:30.333375 2609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:35:30.334307 2609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:35:30.335533 2609 hybrid_clock.cc:648] HybridClock initialized: now 1790408130335497 us; error 56 us; skew 500 ppm
I20260926 07:35:30.335836 2609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:35:30 dist-test-slave-qlq8 krb5kdc[1771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408130, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:35:30.338876 2609 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:35:30.340315 2609 webserver.cc:460] Webserver started at http://127.22.221.67:32939/ using document root <none> and password file <none>
I20260926 07:35:30.341053 2609 fs_manager.cc:359] Metadata directory not provided
I20260926 07:35:30.341193 2609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:35:30.341497 2609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:35:30 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408130, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:35:30.356418 2609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "2ffc3b6a21f34ceabee6b02f541ad587"
format_stamp: "Formatted at 2026-09-26 07:35:30 on dist-test-slave-qlq8"
server_key: "69ac6f6f1e8b362ae16287395fef16f8"
server_key_iv: "57568baad7de1e7b8e0fb32452e628ac"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:30.357283 2609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "2ffc3b6a21f34ceabee6b02f541ad587"
format_stamp: "Formatted at 2026-09-26 07:35:30 on dist-test-slave-qlq8"
server_key: "69ac6f6f1e8b362ae16287395fef16f8"
server_key_iv: "57568baad7de1e7b8e0fb32452e628ac"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:30.362506 2609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:35:30.379361 2624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:30.381165 2609 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:35:30.381330 2609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "2ffc3b6a21f34ceabee6b02f541ad587"
format_stamp: "Formatted at 2026-09-26 07:35:30 on dist-test-slave-qlq8"
server_key: "69ac6f6f1e8b362ae16287395fef16f8"
server_key_iv: "57568baad7de1e7b8e0fb32452e628ac"
server_key_version: "kuduclusterkey@0"
I20260926 07:35:30.381577 2609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:35:30.406390 2609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:35:30.409795 2609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:35:30.410056 2609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:35:30.410919 2609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:35:30.412269 2609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:35:30.412340 2609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:30.412418 2609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:35:30.412460 2609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:35:30.445030 2609 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:44565
I20260926 07:35:30.445073 2736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:44565 every 8 connection(s)
I20260926 07:35:30.446333 2609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:35:30.451756 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 2609
I20260926 07:35:30.451938 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:35:30.452409 23413 external_mini_cluster.cc:1428] Setting key 4386454534a11c00cb48ad1375c53cd2
Sep 26 07:35:30 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408130, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:35:30.467971 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.448941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:36367 (local address 127.22.221.126:33197)
0926 07:35:30.449424 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.449428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.450403 (+ 975us) server_negotiation.cc:407] Connection header received
0926 07:35:30.452389 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.452395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.452524 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.452665 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:30.454142 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.454899 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.456579 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.457113 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.457235 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.460558 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:30.460566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:30.460574 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:30.460619 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:30.463642 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:30.464757 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:30.464762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:30.464767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:30.464929 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:30.465511 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:30.465517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:30.465523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:30.465915 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:30.466083 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.467312 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.467544 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260926 07:35:30.469787 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.449518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:33197 (local address 127.22.221.67:36367)
0926 07:35:30.450213 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:30.450249 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:30.452102 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:30.452934 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:30.452949 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:30.453398 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:35:30.453981 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.453998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.455048 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.455058 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.456440 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.456452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.457357 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.457364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.457588 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.458224 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:35:30.458252 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:35:30.460326 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:35:30.463825 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:35:30.463833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:35:30.463852 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:35:30.464591 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:35:30.465045 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:35:30.465050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:35:30.465053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:35:30.465358 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:35:30.466134 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:35:30.466145 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:35:30.466604 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 07:35:30.466909 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 07:35:30.468543 (+ 1634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":295,"threads_started":1}
I20260926 07:35:30.471026 2737 heartbeater.cc:347] Connected to a master server at 127.22.221.126:33197
I20260926 07:35:30.471355 2737 heartbeater.cc:464] Registering TS with master...
I20260926 07:35:30.472046 2737 heartbeater.cc:511] Master 127.22.221.126:33197 requested a full tablet report, sending...
I20260926 07:35:30.473361 2247 ts_manager.cc:194] Registered new tserver with Master: 2ffc3b6a21f34ceabee6b02f541ad587 (127.22.221.67:44565)
I20260926 07:35:30.474166 2247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:36367
I20260926 07:35:30.480257 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:35:30.496665 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.484757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.22.221.126:33197)
0926 07:35:30.484927 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.484931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.485346 (+ 415us) server_negotiation.cc:407] Connection header received
0926 07:35:30.485497 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.485500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.485565 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.485643 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:30.487080 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.487875 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.489198 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.489586 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.489673 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.491919 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:30.491923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:30.491926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:30.491953 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:30.494394 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:30.494949 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:30.494953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:30.494955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:30.495066 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:30.495474 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:30.495478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:30.495480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:30.495736 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:30.495957 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.496279 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.496452 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:35:30.500422 2247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:35:30.507207 2247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:35:30.529690 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.518976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:44565 (local address 127.0.0.1:49854)
0926 07:35:30.520264 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:30.520306 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:30.520655 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:30.521742 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:30.521750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:30.521800 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:35:30.522275 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.522288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.524622 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.524626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.526986 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.527006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.528670 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.528676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.528856 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.528912 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 07:35:30.529065 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:35:30.529242 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":24,"spinlock_wait_cycles":34304,"thread_start_us":875,"threads_started":1}
I20260926 07:35:30.529997 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.519018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:34837 (local address 127.0.0.1:56802)
0926 07:35:30.521729 (+ 2711us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:30.521753 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:30.521901 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:30.522572 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:30.522576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:30.522600 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:35:30.522980 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.522994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.525229 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.525235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.527137 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.527150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.529017 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.529026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.529183 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.529213 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:35:30.529299 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:35:30.529443 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1914,"mutex_wait_us":9,"thread_start_us":1757,"threads_started":1}
I20260926 07:35:30.530229 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.519632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.22.221.65:34837)
0926 07:35:30.520621 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.520629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.522067 (+ 1438us) server_negotiation.cc:407] Connection header received
0926 07:35:30.522168 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.522174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.522357 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.522583 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:35:30.523164 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.525079 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.527324 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.528732 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.528850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.529086 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.529363 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.529539 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":652,"threads_started":1}
I20260926 07:35:30.530637 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.519126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.22.221.67:44565)
0926 07:35:30.519838 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.519850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.520428 (+ 578us) server_negotiation.cc:407] Connection header received
0926 07:35:30.520945 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.520954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.521206 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.521362 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:35:30.522489 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.524463 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.527162 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.528481 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.528635 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.528861 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.530147 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.530326 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":380,"threads_started":1}
I20260926 07:35:30.530689 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.521130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:42927 (local address 127.0.0.1:58306)
0926 07:35:30.521685 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:30.521705 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:30.521810 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:30.522884 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:30.522888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:30.522916 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:35:30.523342 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.523354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.525186 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.525192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.527883 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.527896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.529857 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.529864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.530037 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.530071 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:35:30.530159 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:35:30.530292 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":20,"spinlock_wait_cycles":6784,"thread_start_us":318,"threads_started":1}
I20260926 07:35:30.530951 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.521338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.22.221.66:42927)
0926 07:35:30.522295 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.522304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.522333 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:35:30.522438 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.522447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.522669 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.522883 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:35:30.523533 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.525043 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.528058 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.529388 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.529524 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.529694 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.530455 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.530610 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":693,"threads_started":1}
I20260926 07:35:30.534005 2672 tablet_service.cc:1467] Processing CreateTablet for tablet af948f7e063c47f19f028d2155d8ac7e (DEFAULT_TABLE table=test-table [id=c38d4d2f848d416e9152026c5451b287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:35:30.534245 2395 tablet_service.cc:1467] Processing CreateTablet for tablet af948f7e063c47f19f028d2155d8ac7e (DEFAULT_TABLE table=test-table [id=c38d4d2f848d416e9152026c5451b287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:35:30.534401 2538 tablet_service.cc:1467] Processing CreateTablet for tablet af948f7e063c47f19f028d2155d8ac7e (DEFAULT_TABLE table=test-table [id=c38d4d2f848d416e9152026c5451b287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:35:30.535460 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af948f7e063c47f19f028d2155d8ac7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:35:30.535673 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af948f7e063c47f19f028d2155d8ac7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:35:30.535692 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af948f7e063c47f19f028d2155d8ac7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:35:30.547554 2756 tablet_bootstrap.cc:492] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Bootstrap starting.
I20260926 07:35:30.549993 2757 tablet_bootstrap.cc:492] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Bootstrap starting.
I20260926 07:35:30.550743 2758 tablet_bootstrap.cc:492] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Bootstrap starting.
I20260926 07:35:30.551185 2756 tablet_bootstrap.cc:654] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Neither blocks nor log segments found. Creating new log.
I20260926 07:35:30.552515 2756 log.cc:824] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Log is configured to *not* fsync() on all Append() calls
I20260926 07:35:30.554164 2757 tablet_bootstrap.cc:654] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Neither blocks nor log segments found. Creating new log.
I20260926 07:35:30.554387 2758 tablet_bootstrap.cc:654] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Neither blocks nor log segments found. Creating new log.
I20260926 07:35:30.555198 2757 log.cc:824] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:35:30.555739 2758 log.cc:824] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Log is configured to *not* fsync() on all Append() calls
I20260926 07:35:30.558030 2756 tablet_bootstrap.cc:492] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: No bootstrap required, opened a new log
I20260926 07:35:30.558311 2756 ts_tablet_manager.cc:1397] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260926 07:35:30.558513 2757 tablet_bootstrap.cc:492] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: No bootstrap required, opened a new log
I20260926 07:35:30.558768 2757 ts_tablet_manager.cc:1397] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 07:35:30.558995 2758 tablet_bootstrap.cc:492] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: No bootstrap required, opened a new log
I20260926 07:35:30.559291 2758 ts_tablet_manager.cc:1397] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:35:30.566819 2758 raft_consensus.cc:348] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.567198 2758 raft_consensus.cc:374] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:35:30.567365 2758 raft_consensus.cc:729] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83cf9ed308142bf9de9a0412e707460, State: Initialized, Role: FOLLOWER
I20260926 07:35:30.568068 2758 consensus_queue.cc:260] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.569339 2603 heartbeater.cc:503] Master 127.22.221.126:33197 was elected leader, sending a full tablet report...
I20260926 07:35:30.569167 2757 raft_consensus.cc:348] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.569121 2756 raft_consensus.cc:348] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.569562 2757 raft_consensus.cc:374] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:35:30.569561 2756 raft_consensus.cc:374] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:35:30.569664 2756 raft_consensus.cc:729] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ffc3b6a21f34ceabee6b02f541ad587, State: Initialized, Role: FOLLOWER
I20260926 07:35:30.569664 2757 raft_consensus.cc:729] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd0398ea1d347deab0d0580d15ddb0a, State: Initialized, Role: FOLLOWER
I20260926 07:35:30.570235 2757 consensus_queue.cc:260] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.570233 2756 consensus_queue.cc:260] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.571620 2758 ts_tablet_manager.cc:1428] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Time spent starting tablet: real 0.012s user 0.008s sys 0.002s
I20260926 07:35:30.574826 2737 heartbeater.cc:503] Master 127.22.221.126:33197 was elected leader, sending a full tablet report...
I20260926 07:35:30.575215 2757 ts_tablet_manager.cc:1428] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Time spent starting tablet: real 0.016s user 0.013s sys 0.005s
I20260926 07:35:30.575449 2756 ts_tablet_manager.cc:1428] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260926 07:35:30.629755 2604 tablet.cc:2369] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:35:30.671339 2461 tablet.cc:2369] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:35:30.697969 2738 tablet.cc:2369] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:35:30.731817 2762 raft_consensus.cc:482] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:35:30.732087 2762 raft_consensus.cc:504] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.733824 2762 leader_election.cc:288] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ffc3b6a21f34ceabee6b02f541ad587 (127.22.221.67:44565), 9bd0398ea1d347deab0d0580d15ddb0a (127.22.221.65:34837)
I20260926 07:35:30.741065 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.734276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:34381 (local address 127.22.221.67:44565)
0926 07:35:30.734477 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.734483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.735467 (+ 984us) server_negotiation.cc:407] Connection header received
0926 07:35:30.735612 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.735615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.735670 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.735774 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:35:30.736481 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.737191 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.739164 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.740037 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.740220 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.740290 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.740648 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.740803 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:35:30.741149 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.734416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:43107 (local address 127.22.221.65:34837)
0926 07:35:30.734652 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.734661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.735321 (+ 660us) server_negotiation.cc:407] Connection header received
0926 07:35:30.735568 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.735577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.735657 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.735733 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:35:30.736573 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.737326 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.739084 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.740013 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.740164 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.740250 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.740781 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.740920 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:35:30.741231 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.734311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:44565 (local address 127.22.221.66:34381)
0926 07:35:30.735343 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:30.735361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:30.735509 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:30.735806 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:30.735810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:30.735832 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:35:30.736309 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.736319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.737333 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.737338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.738977 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.739016 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.740285 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.740291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.740442 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.740465 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:35:30.740588 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:35:30.740691 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":129,"thread_start_us":482,"threads_started":1}
I20260926 07:35:30.741797 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.734312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:34837 (local address 127.22.221.66:43107)
0926 07:35:30.735176 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 07:35:30.735191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:35:30.735386 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:35:30.735779 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:35:30.735783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:35:30.735806 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:35:30.736380 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.736412 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.737468 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.737471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.738921 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:35:30.738932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.740246 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.740253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:35:30.740488 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.740633 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 07:35:30.740755 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:35:30.740866 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":38,"thread_start_us":523,"threads_started":1}
I20260926 07:35:30.742501 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af948f7e063c47f19f028d2155d8ac7e" candidate_uuid: "f83cf9ed308142bf9de9a0412e707460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" is_pre_election: true
I20260926 07:35:30.742744 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af948f7e063c47f19f028d2155d8ac7e" candidate_uuid: "f83cf9ed308142bf9de9a0412e707460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" is_pre_election: true
I20260926 07:35:30.742982 2692 raft_consensus.cc:2454] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83cf9ed308142bf9de9a0412e707460 in term 0.
I20260926 07:35:30.743292 2415 raft_consensus.cc:2454] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83cf9ed308142bf9de9a0412e707460 in term 0.
I20260926 07:35:30.743831 2492 leader_election.cc:302] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ffc3b6a21f34ceabee6b02f541ad587, f83cf9ed308142bf9de9a0412e707460; no voters:
I20260926 07:35:30.744266 2762 raft_consensus.cc:2781] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:35:30.744410 2762 raft_consensus.cc:482] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:35:30.744535 2762 raft_consensus.cc:3037] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:35:30.746666 2762 raft_consensus.cc:504] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.747282 2762 leader_election.cc:288] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [CANDIDATE]: Term 1 election: Requested vote from peers 2ffc3b6a21f34ceabee6b02f541ad587 (127.22.221.67:44565), 9bd0398ea1d347deab0d0580d15ddb0a (127.22.221.65:34837)
I20260926 07:35:30.747677 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af948f7e063c47f19f028d2155d8ac7e" candidate_uuid: "f83cf9ed308142bf9de9a0412e707460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ffc3b6a21f34ceabee6b02f541ad587"
I20260926 07:35:30.747727 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af948f7e063c47f19f028d2155d8ac7e" candidate_uuid: "f83cf9ed308142bf9de9a0412e707460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd0398ea1d347deab0d0580d15ddb0a"
I20260926 07:35:30.747875 2692 raft_consensus.cc:3037] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:35:30.747900 2415 raft_consensus.cc:3037] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:35:30.749927 2692 raft_consensus.cc:2454] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83cf9ed308142bf9de9a0412e707460 in term 1.
I20260926 07:35:30.749962 2415 raft_consensus.cc:2454] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83cf9ed308142bf9de9a0412e707460 in term 1.
I20260926 07:35:30.750339 2494 leader_election.cc:302] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ffc3b6a21f34ceabee6b02f541ad587, 9bd0398ea1d347deab0d0580d15ddb0a, f83cf9ed308142bf9de9a0412e707460; no voters:
I20260926 07:35:30.750620 2762 raft_consensus.cc:2781] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:35:30.751060 2762 raft_consensus.cc:686] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 1 LEADER]: Becoming Leader. State: Replica: f83cf9ed308142bf9de9a0412e707460, State: Running, Role: LEADER
I20260926 07:35:30.751555 2762 consensus_queue.cc:237] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } }
I20260926 07:35:30.756094 2247 catalog_manager.cc:5495] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 reported cstate change: term changed from 0 to 1, leader changed from <none> to f83cf9ed308142bf9de9a0412e707460 (127.22.221.66). New cstate: current_term: 1 leader_uuid: "f83cf9ed308142bf9de9a0412e707460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83cf9ed308142bf9de9a0412e707460" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42927 } health_report { overall_health: HEALTHY } } }
I20260926 07:35:30.828635 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.823142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.22.221.66:42927)
0926 07:35:30.823431 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.823436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.823456 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:35:30.823694 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.823698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.823783 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.823938 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:35:30.824338 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.825055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.826467 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.826870 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.827001 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.827565 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:35:30.828053 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:35:30.828212 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.828272 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.828347 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:35:30.836546 2692 raft_consensus.cc:1261] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 1 FOLLOWER]: Refusing update from remote peer f83cf9ed308142bf9de9a0412e707460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:35:30.836548 2415 raft_consensus.cc:1261] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 1 FOLLOWER]: Refusing update from remote peer f83cf9ed308142bf9de9a0412e707460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:35:30.837499 2767 consensus_queue.cc:1035] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ffc3b6a21f34ceabee6b02f541ad587" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:35:30.837823 2762 consensus_queue.cc:1035] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bd0398ea1d347deab0d0580d15ddb0a" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:35:30.849398 2770 mvcc.cc:205] Tried to move back new op lower bound from 7333511703897284608 to 7333511703561830400. Current Snapshot: MvccSnapshot[applied={T|T < 7333511703897284608}]
I20260926 07:35:30.851002 2771 mvcc.cc:205] Tried to move back new op lower bound from 7333511703897284608 to 7333511703561830400. Current Snapshot: MvccSnapshot[applied={T|T < 7333511703897284608}]
I20260926 07:35:30.865365 2246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:35:30.865715 2246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:35:30.870062 2246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6791bd67a4f54460901b33ccd2db9957: Sending DeleteTablet for 3 replicas of tablet af948f7e063c47f19f028d2155d8ac7e
I20260926 07:35:30.871254 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet af948f7e063c47f19f028d2155d8ac7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:35:30 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:56802
I20260926 07:35:30.871249 2672 tablet_service.cc:1514] Processing DeleteTablet for tablet af948f7e063c47f19f028d2155d8ac7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:35:30 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49854
I20260926 07:35:30.874307 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet af948f7e063c47f19f028d2155d8ac7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:35:30 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:58306
I20260926 07:35:30.884639 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 2327
I20260926 07:35:30.897085 2782 tablet_replica.cc:330] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: stopping tablet replica
I20260926 07:35:30.897544 2782 raft_consensus.cc:2229] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:35:30.898727 2780 tablet_replica.cc:330] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: stopping tablet replica
I20260926 07:35:30.899186 2780 raft_consensus.cc:2229] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:35:30.899803 2780 raft_consensus.cc:2258] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:35:30.900563 2782 raft_consensus.cc:2258] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:35:30.900892 2781 tablet_replica.cc:330] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: stopping tablet replica
I20260926 07:35:30.901293 2781 raft_consensus.cc:2229] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:35:30.901751 2781 raft_consensus.cc:2258] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:35:30.902505 2782 ts_tablet_manager.cc:1905] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:35:30.902676 2780 ts_tablet_manager.cc:1905] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:35:30.903772 2781 ts_tablet_manager.cc:1905] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:35:30 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:35:30.909544 2780 ts_tablet_manager.cc:1918] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:35:30.909647 2782 ts_tablet_manager.cc:1918] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:35:30.909693 2780 log.cc:1196] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/af948f7e063c47f19f028d2155d8ac7e
I20260926 07:35:30.909801 2782 log.cc:1196] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/af948f7e063c47f19f028d2155d8ac7e
I20260926 07:35:30.910207 2780 ts_tablet_manager.cc:1939] T af948f7e063c47f19f028d2155d8ac7e P 9bd0398ea1d347deab0d0580d15ddb0a: Deleting consensus metadata
I20260926 07:35:30.910252 2781 ts_tablet_manager.cc:1918] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:35:30.910326 2782 ts_tablet_manager.cc:1939] T af948f7e063c47f19f028d2155d8ac7e P f83cf9ed308142bf9de9a0412e707460: Deleting consensus metadata
I20260926 07:35:30.910385 2781 log.cc:1196] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/af948f7e063c47f19f028d2155d8ac7e
I20260926 07:35:30.910840 2781 ts_tablet_manager.cc:1939] T af948f7e063c47f19f028d2155d8ac7e P 2ffc3b6a21f34ceabee6b02f541ad587: Deleting consensus metadata
I20260926 07:35:30.912220 2234 catalog_manager.cc:4841] TS 9bd0398ea1d347deab0d0580d15ddb0a (127.22.221.65:34837): tablet af948f7e063c47f19f028d2155d8ac7e (table test-table [id=c38d4d2f848d416e9152026c5451b287]) successfully deleted
I20260926 07:35:30.912424 2232 catalog_manager.cc:4841] TS 2ffc3b6a21f34ceabee6b02f541ad587 (127.22.221.67:44565): tablet af948f7e063c47f19f028d2155d8ac7e (table test-table [id=c38d4d2f848d416e9152026c5451b287]) successfully deleted
I20260926 07:35:30.913211 2234 catalog_manager.cc:4841] TS f83cf9ed308142bf9de9a0412e707460 (127.22.221.66:42927): tablet af948f7e063c47f19f028d2155d8ac7e (table test-table [id=c38d4d2f848d416e9152026c5451b287]) successfully deleted
I20260926 07:35:30.918357 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:30.885738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.22.221.65:34837)
0926 07:35:30.885922 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:30.885927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:30.898132 (+ 12205us) server_negotiation.cc:407] Connection header received
0926 07:35:30.898328 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:30.898334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:30.898435 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:30.898560 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:30.900928 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.902042 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.903893 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:30.904535 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:30.904643 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:30.910843 (+ 6200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:30.910859 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:30.910876 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:30.910920 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:30.915445 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:30.916138 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:30.916147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:30.916151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:30.916308 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:30.916723 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:30.916729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:30.916732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:30.917154 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:30.917447 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:30.917746 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 07:35:30.917990 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:35:30.919040 2455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:35:31.008476 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 2327
I20260926 07:35:31.026310 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 2466
Sep 26 07:35:31 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:35:31.042447 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:31.027140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.22.221.66:42927)
0926 07:35:31.027341 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:31.027347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:31.027447 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:35:31.027626 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:31.027631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:31.027743 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:31.027881 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:31.029120 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:31.029977 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:31.031565 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:31.031989 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:31.032151 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:31.036397 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:31.036418 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:31.036437 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:31.036473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:31.039618 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:31.040307 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:31.040317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:31.040320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:31.040460 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:31.040866 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:31.040869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:31.040872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:31.041321 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:31.041603 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:31.041979 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 07:35:31.042141 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:35:31.043143 2598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:35:31.159020 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 2466
I20260926 07:35:31.177313 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 2609
Sep 26 07:35:31 dist-test-slave-qlq8 krb5kdc[1771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:35:31.192024 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:35:31.178130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.22.221.67:44565)
0926 07:35:31.178313 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:35:31.178317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:35:31.178335 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:35:31.178492 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:35:31.178495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:35:31.178556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:35:31.178641 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:35:31.179936 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:31.180635 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:31.181991 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:35:31.182350 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:35:31.182447 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:35:31.186625 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:35:31.186638 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:35:31.186653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:35:31.186689 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:35:31.189514 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:31.190124 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:31.190131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:31.190133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:31.190275 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:35:31.190571 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:35:31.190574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:35:31.190576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:35:31.190919 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:35:31.191124 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:35:31.191513 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 07:35:31.191661 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:35:31.192682 2732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:35:31.281390 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 2609
I20260926 07:35:31.297636 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 2215
I20260926 07:35:31.298820 2277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:35:31.410389 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 2215
I20260926 07:35:31.434494 23413 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:35:31.773087 23413 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:35:31.773331 23413 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:35:32.118947 23413 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:35:32.119 UTC [1801] LOG: received smart shutdown request
2026-09-26 07:35:32.122 UTC [1801] LOG: background worker "logical replication launcher" (PID 1810) exited with exit code 1
2026-09-26 07:35:32.122 UTC [1805] LOG: shutting down
2026-09-26 07:35:32.286 UTC [1801] LOG: database system is shut down
2026-09-26T07:35:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48800 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:35:32 dist-test-slave-qlq8 krb5kdc[2796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:35:32 dist-test-slave-qlq8 krb5kdc[2796](info): set up 2 sockets
Sep 26 07:35:32 dist-test-slave-qlq8 krb5kdc[2796](info): commencing operation
krb5kdc: starting...
W20260926 07:35:33.531735 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:35:33 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:35:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/rangeradmin_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/rangeradmin_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/rangerlookup_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/rangerlookup_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
I20260926 07:35:33.858013 23413 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdo7xD1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/postgres -l logfile start
2026-09-26 07:35:36.306 UTC [2826] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:35:36.306 UTC [2826] LOG: listening on IPv4 address "127.22.221.84", port 33183
2026-09-26 07:35:36.309 UTC [2826] LOG: listening on Unix socket "/tmp/.s.PGSQL.33183"
2026-09-26 07:35:36.332 UTC [2829] LOG: database system was shut down at 2026-09-26 07:35:34 UTC
2026-09-26 07:35:36.336 UTC [2826] LOG: database system is ready to accept connections
I20260926 07:35:37.311503 23413 mini_postgres.cc:96] Postgres bound to 33183
2026-09-26 07:35:37.332 UTC [2837] FATAL: database "slave" does not exist
127.22.221.84:33183 - accepting connections
I20260926 07:35:37.333525 23413 mini_ranger.cc:161] Starting Ranger...
I20260926 07:35:37.358503 23413 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:35:37.830543 23413 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:35:37.830662 23413 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-admin
2026-09-26 07:35:38,134 [I] DB FLAVOR :POSTGRES
2026-09-26 07:35:38,134 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:35:38,134 [I] Checking connection..
2026-09-26 07:35:38,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:35:38,395 [I] Checking connection passed.
2026-09-26 07:35:38,395 [I] --------- Verifying version history table ---------
2026-09-26 07:35:38,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:35:38,629 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:35:38,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:35:38,881 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:35:38,881 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:35:38,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:35:39.139 UTC [2901] WARNING: there is no transaction in progress
2026-09-26 07:35:39,145 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:35:39,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:35:39,403 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:35:39,403 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:35:39,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:35:39,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:35:39,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qlq8.c.gcp-upstream.internal','N') ;"
2026-09-26 07:35:40,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:35:40,337 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:35:40,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:35:40,567 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:35:40,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:35:40,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:35:41,079 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:35:41,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:35:41.994 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.010 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.025 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.042 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.060 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.078 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.134 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.141 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.155 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.165 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.175 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.188 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.197 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.203 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.210 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42.216 UTC [3009] WARNING: there is no transaction in progress
2026-09-26 07:35:42,845 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:35:42,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:35:43,098 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:35:43,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:35:43,353 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:35:43,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:35:43,613 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:35:43,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:35:43,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:35:44,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qlq8.c.gcp-upstream.internal';"
2026-09-26 07:35:44,325 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:35:44,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:35:44,567 [I] DB_PATCHES have already been applied
I20260926 07:35:44.573609 23413 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdo7xD1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:35:44.573690 23413 mini_ranger.cc:193] Using host: 127.22.221.84
I20260926 07:35:44.588860 23413 mini_ranger.cc:227] Ranger admin URL: http://127.22.221.84:41285
Sep 26, 2026 7:35:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:35:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:35:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:35:45.709795 23413 mini_ranger.cc:160] Time spent starting Ranger: real 8.376s user 0.007s sys 0.031s
Sep 26, 2026 7:36:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:36:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:36:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:36:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:36:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:36:08.022277 23413 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/rangerkms_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/rangerkms_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.221.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.221.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.221.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/HTTP_127.22.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:36:08 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408168, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:36:08.184267 23413 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:36:08.205377 23413 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:36:09.067420 23413 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:36:09.067526 23413 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms
I20260926 07:36:09.067580 23413 mini_ranger_kms.cc:227] Using postgres at 127.22.221.84:33183
2026-09-26 07:36:09,412 [I] DB FLAVOR :POSTGRES
2026-09-26 07:36:09,412 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:36:09,413 [I] Checking connection
2026-09-26 07:36:09,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:36:09,639 [I] connection success
2026-09-26 07:36:09,639 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:36:09,639 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:36:09,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:36:09,869 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:36:09,870 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:36:09,870 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:36:09,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.221.84:33183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:36:10,127 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:36:10.610764 23413 mini_ranger_kms.cc:310] Created kms service
I20260926 07:36:10.780510 23413 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:36:10.825481 23413 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:36:10 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408168, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:11.204182 23413 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:36:11.204510 23413 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdo7xD1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdo7xD1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:36:11.204558 23413 mini_ranger_kms.cc:240] Using host: 127.22.221.84
I20260926 07:36:11.225605 23413 mini_ranger_kms.cc:276] Ranger KMS PID: 3189
I20260926 07:36:11.225751 23413 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.221.84:34541
Sep 26, 2026 7:36:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:36:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:36:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:36:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:36:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:36:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:36:13.539350 23413 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.355s user 0.067s sys 0.292s
I20260926 07:36:13.539608 23413 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:36:13.539788 23413 mini_ranger_kms.cc:401] 127.22.221.84:34541/kms/v1/keys
Sep 26 07:36:13 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790408173, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.221.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:36:14 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408173, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.221.84@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
Sep 26 07:36:14 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408173, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.221.84@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
Sep 26 07:36:16 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:36:16.653088 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39209
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:39209
--ranger_config_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:16.980690 3240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:16.981053 3240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:16.981227 3240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:16.988734 3240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:16.988883 3240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:16.989003 3240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:16.989108 3240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:16.989198 3240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:16.999744 3240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.221.84:34541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:39209
--ranger_config_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:17.001150 3240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:17.002583 3240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:17.015489 3245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:17.015429 3248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:17.015535 3246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:17.016564 3240 server_base.cc:1034] running on GCE node
I20260926 07:36:17.017336 3240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:17.018692 3240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:17.019943 3240 hybrid_clock.cc:648] HybridClock initialized: now 1790408177019899 us; error 68 us; skew 500 ppm
I20260926 07:36:17.020301 3240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:17 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408177, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:17.028337 3240 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:17.029817 3240 webserver.cc:460] Webserver started at http://127.22.221.126:34405/ using document root <none> and password file <none>
I20260926 07:36:17.030473 3240 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:17.030629 3240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:17.030936 3240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:36:17 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408177, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:17.090870 3240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd"
format_stamp: "Formatted at 2026-09-26 07:36:17 on dist-test-slave-qlq8"
server_key: "892ca1172cce6c2efd5bf2ae5245ef94"
server_key_iv: "f255af75969383a4b02b2814c3b9ac6e"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:17.091771 3240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd"
format_stamp: "Formatted at 2026-09-26 07:36:17 on dist-test-slave-qlq8"
server_key: "892ca1172cce6c2efd5bf2ae5245ef94"
server_key_iv: "f255af75969383a4b02b2814c3b9ac6e"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:17.097081 3240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:36:17.113332 3255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:17.115069 3240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:36:17.115303 3240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd"
format_stamp: "Formatted at 2026-09-26 07:36:17 on dist-test-slave-qlq8"
server_key: "892ca1172cce6c2efd5bf2ae5245ef94"
server_key_iv: "f255af75969383a4b02b2814c3b9ac6e"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:17.115552 3240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:17.148627 3240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:17.152452 3240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:17.152776 3240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:17.171546 3240 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:39209
I20260926 07:36:17.171581 3306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:39209 every 8 connection(s)
I20260926 07:36:17.172974 3240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:17.175971 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 3240
I20260926 07:36:17.176201 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:36:17.176899 23413 external_mini_cluster.cc:1428] Setting key a3068b3d06e44604d771d884786fc5be
I20260926 07:36:17.177043 3307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:17.191322 3307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Bootstrap starting.
Sep 26 07:36:17 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:17.195096 3307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:17.196442 3307 log.cc:824] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:17.199930 3307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: No bootstrap required, opened a new log
I20260926 07:36:17.201467 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:17.178891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.22.221.126:39209)
0926 07:36:17.179677 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:17.179687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:17.179731 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:36:17.181615 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:17.181636 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:17.182021 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:17.182338 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:17.184108 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:17.185477 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:17.187345 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:17.188068 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:17.188194 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:17.192879 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:17.192889 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:17.192907 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:17.192959 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:17.196781 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:17.197750 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:17.197756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:17.197762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:17.197981 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:17.198382 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:17.198386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:17.198389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:17.199053 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:17.199377 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:17.199778 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 07:36:17.200178 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":394,"threads_started":1}
I20260926 07:36:17.207646 3307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:17.207955 3307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:17.208027 3307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb86cf4f3d74649ae4e57840ba6ebcd, State: Initialized, Role: FOLLOWER
I20260926 07:36:17.208653 3307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:17.208916 3307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:17.209103 3307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:17.209275 3307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:17.211431 3307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:17.211923 3307 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb86cf4f3d74649ae4e57840ba6ebcd; no voters:
I20260926 07:36:17.212785 3307 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:36:17.213023 3312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:17.213822 3312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 LEADER]: Becoming Leader. State: Replica: 6fb86cf4f3d74649ae4e57840ba6ebcd, State: Running, Role: LEADER
I20260926 07:36:17.214327 3312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:17.215134 3307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:17.217017 3313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } } }
I20260926 07:36:17.217658 3313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:17.217885 3314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb86cf4f3d74649ae4e57840ba6ebcd. Latest consensus state: current_term: 1 leader_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } } }
I20260926 07:36:17.218459 3314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:17.220335 3317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:17.224895 3317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:17.228011 3307 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:36:17.239331 3317 catalog_manager.cc:1347] Generated new cluster ID: ca74d5602f1342518624290c5c1985bb
I20260926 07:36:17.239532 3317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:17.255268 3317 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:36:17.257186 3317 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:17.272740 3317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Generated new TSK 0
I20260926 07:36:17.274051 3317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:36:18.044771 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:39209
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:36:18 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790408178, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:36:18.506696 3352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:18.507066 3352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:18.507277 3352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:18.515736 3352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:18.515894 3352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:18.516080 3352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:36:18.528090 3352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.221.84:34541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:39209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:18.529524 3352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:18.531180 3352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:18.555682 3362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:18.557155 3363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:18.565333 3365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:18.565819 3352 server_base.cc:1034] running on GCE node
I20260926 07:36:18.566694 3352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:18.568110 3352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:18.569392 3352 hybrid_clock.cc:648] HybridClock initialized: now 1790408178569330 us; error 72 us; skew 500 ppm
I20260926 07:36:18.569804 3352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:18 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408178, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:18.574370 3352 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:36:18.579370 3352 webserver.cc:460] Webserver started at http://127.22.221.65:43131/ using document root <none> and password file <none>
I20260926 07:36:18.580327 3352 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:18.580519 3352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:18.580889 3352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:36:18 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408178, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:18.603801 3352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "1b35bc6bfc224dcab618dec4e0c76c67"
format_stamp: "Formatted at 2026-09-26 07:36:18 on dist-test-slave-qlq8"
server_key: "06525315d45c2ceb839700e35e58c644"
server_key_iv: "9afe4248ebce17fee489cb00f4520493"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:18.605002 3352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "1b35bc6bfc224dcab618dec4e0c76c67"
format_stamp: "Formatted at 2026-09-26 07:36:18 on dist-test-slave-qlq8"
server_key: "06525315d45c2ceb839700e35e58c644"
server_key_iv: "9afe4248ebce17fee489cb00f4520493"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:18.613306 3352 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s
I20260926 07:36:18.646273 3372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:18.648448 3352 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260926 07:36:18.648655 3352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "1b35bc6bfc224dcab618dec4e0c76c67"
format_stamp: "Formatted at 2026-09-26 07:36:18 on dist-test-slave-qlq8"
server_key: "06525315d45c2ceb839700e35e58c644"
server_key_iv: "9afe4248ebce17fee489cb00f4520493"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:18.648929 3352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:18.673552 3352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:18.683410 3352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:18.683821 3352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:18.687951 3352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:18.689592 3352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:18.689740 3352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:18.689944 3352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:18.690062 3352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:18.737285 3352 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:46709
I20260926 07:36:18.737314 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:46709 every 8 connection(s)
I20260926 07:36:18.739145 3352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:36:18.746313 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 3352
I20260926 07:36:18.746568 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:36:18.747452 23413 external_mini_cluster.cc:1428] Setting key 2c78793ffe7606c1a9bd2ac97472ec6e
Sep 26 07:36:18 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408178, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:36:18.812429 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:18.748180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:33039 (local address 127.22.221.126:39209)
0926 07:36:18.782957 (+ 34777us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:18.782967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:18.787313 (+ 4346us) server_negotiation.cc:407] Connection header received
0926 07:36:18.790020 (+ 2707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:18.790034 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:18.790156 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:18.790278 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:18.792670 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:18.793935 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:18.796877 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:18.797697 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:18.797810 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:18.802892 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:18.802901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:18.802909 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:18.802962 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:18.807110 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:18.809557 (+ 2447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:18.809563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:18.809567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:18.809728 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:18.810347 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:18.810353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:18.810356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:18.810741 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:18.810890 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:18.811805 (+ 915us) server_negotiation.cc:299] Negotiation successful
0926 07:36:18.812035 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34563,"thread_start_us":34788,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
I20260926 07:36:18.814394 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:18.747361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39209 (local address 127.22.221.65:33039)
0926 07:36:18.783925 (+ 36564us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:18.783998 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:18.789692 (+ 5694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:18.790782 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:18.790799 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:18.791563 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:18.792452 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:18.792481 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:18.794168 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:18.794176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:18.796680 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:18.796699 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:18.798205 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:18.798215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:18.798570 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:18.799462 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:18.799498 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:18.802602 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:18.808297 (+ 5695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:18.808307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:18.808327 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:18.808995 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:18.809908 (+ 913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:18.809913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:18.809917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:18.810172 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:18.810975 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:18.810981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:18.811385 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 07:36:18.811677 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 07:36:18.812265 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36073,"thread_start_us":39844,"threads_started":1}
I20260926 07:36:18.816629 3485 heartbeater.cc:347] Connected to a master server at 127.22.221.126:39209
I20260926 07:36:18.816977 3485 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:18.817883 3485 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:18.820492 3272 ts_manager.cc:194] Registered new tserver with Master: 1b35bc6bfc224dcab618dec4e0c76c67 (127.22.221.65:46709)
I20260926 07:36:18.822599 3272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:33039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:36:18.847584 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:39209
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:36:18 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408178, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
Sep 26 07:36:19 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408178, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
W20260926 07:36:19.344545 3491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:19.345077 3491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:19.345292 3491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:19.356683 3491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:19.356900 3491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:19.357139 3491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:36:19.372601 3491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.221.84:34541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:39209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:19.374567 3491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:19.376348 3491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:19.411377 3498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:19.413365 3499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:19.419730 3491 server_base.cc:1034] running on GCE node
W20260926 07:36:19.420409 3502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:19.421311 3491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:19.440201 3491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:19.441831 3491 hybrid_clock.cc:648] HybridClock initialized: now 1790408179441789 us; error 59 us; skew 500 ppm
I20260926 07:36:19.442524 3491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:19 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408179, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:19.449728 3491 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:36:19.484802 3491 webserver.cc:460] Webserver started at http://127.22.221.66:45345/ using document root <none> and password file <none>
I20260926 07:36:19.486018 3491 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:19.486258 3491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:19.486711 3491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:36:19 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408179, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:19.514762 3491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "0323636d511641b9aa3423bafd13e895"
format_stamp: "Formatted at 2026-09-26 07:36:19 on dist-test-slave-qlq8"
server_key: "94c136ceb16f7a431f9e5f2abad18e0e"
server_key_iv: "5e3a33b006cc406cb862106332d9282d"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:19.520033 3491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "0323636d511641b9aa3423bafd13e895"
format_stamp: "Formatted at 2026-09-26 07:36:19 on dist-test-slave-qlq8"
server_key: "94c136ceb16f7a431f9e5f2abad18e0e"
server_key_iv: "5e3a33b006cc406cb862106332d9282d"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:19.545266 3491 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.008s sys 0.004s
I20260926 07:36:19.580884 3510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:19.584595 3491 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.008s
I20260926 07:36:19.584856 3491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "0323636d511641b9aa3423bafd13e895"
format_stamp: "Formatted at 2026-09-26 07:36:19 on dist-test-slave-qlq8"
server_key: "94c136ceb16f7a431f9e5f2abad18e0e"
server_key_iv: "5e3a33b006cc406cb862106332d9282d"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:19.585176 3491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:19.607856 3491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:19.612238 3491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:19.612556 3491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:19.613626 3491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:19.615445 3491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:19.615548 3491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:19.615702 3491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:19.615777 3491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:19.666476 3491 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:40967
I20260926 07:36:19.666500 3627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:40967 every 8 connection(s)
I20260926 07:36:19.667904 3491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:36:19.669121 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 3491
I20260926 07:36:19.669337 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:36:19.670053 23413 external_mini_cluster.cc:1428] Setting key beeb1ce49b45506935b4750090fba424
Sep 26 07:36:19 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408179, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:19.701099 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:19.674528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:56345 (local address 127.22.221.126:39209)
0926 07:36:19.682837 (+ 8309us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:19.682843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:19.682873 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:36:19.682932 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:19.682936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:19.683026 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:19.683132 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:19.685341 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:19.686450 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:19.688780 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:19.689372 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:19.689496 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:19.693961 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:19.693968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:19.693973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:19.694013 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:19.697420 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:19.698604 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:19.698610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:19.698614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:19.698746 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:19.699223 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:19.699254 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:19.699259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:19.699592 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:19.699754 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:19.700472 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 07:36:19.700716 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8190,"thread_start_us":8091,"threads_started":1}
I20260926 07:36:19.702225 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:19.674006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39209 (local address 127.22.221.66:56345)
0926 07:36:19.679884 (+ 5878us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:19.679933 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:19.682181 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:19.683589 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:19.683601 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:19.684216 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:19.685155 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:19.685181 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:19.686628 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:19.686634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:19.688645 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:19.688659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:19.689643 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:19.689651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:19.689961 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:19.690771 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:19.690800 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:19.693708 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:19.697602 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:19.697613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:19.697631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:19.698453 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:19.698864 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:19.698870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:19.698884 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:19.699091 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:19.699721 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:19.699730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:19.700088 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 07:36:19.700362 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 07:36:19.700955 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5561,"thread_start_us":5335,"threads_started":1}
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:36:19.703974 3628 heartbeater.cc:347] Connected to a master server at 127.22.221.126:39209
I20260926 07:36:19.704288 3628 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:19.705075 3628 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:19.706727 3272 ts_manager.cc:194] Registered new tserver with Master: 0323636d511641b9aa3423bafd13e895 (127.22.221.66:40967)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
I20260926 07:36:19.707901 3272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:56345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:36:19.738569 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:39209
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:19.826670 3485 heartbeater.cc:503] Master 127.22.221.126:39209 was elected leader, sending a full tablet report...
W20260926 07:36:20.115281 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:20.115571 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:20.115759 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:20.124099 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:20.124215 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:20.124435 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:36:20.134354 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.221.84:34541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:39209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:20.135720 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:20.137029 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:20.149664 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:20.149696 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:20.149775 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:20.153664 3634 server_base.cc:1034] running on GCE node
I20260926 07:36:20.154345 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:20.155455 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:20.156692 3634 hybrid_clock.cc:648] HybridClock initialized: now 1790408180156636 us; error 54 us; skew 500 ppm
I20260926 07:36:20.157013 3634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:20 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408180, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:20.160130 3634 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:36:20.161659 3634 webserver.cc:460] Webserver started at http://127.22.221.67:37107/ using document root <none> and password file <none>
I20260926 07:36:20.162309 3634 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:20.162405 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:20.162645 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:36:20 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408180, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:20.178457 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "a398eb0140ae46e88f500999e5ea3541"
format_stamp: "Formatted at 2026-09-26 07:36:20 on dist-test-slave-qlq8"
server_key: "ddd07c89a01721b410698930bae545a6"
server_key_iv: "cac9ca97204c8d4f7306f2a124e42804"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:20.179476 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "a398eb0140ae46e88f500999e5ea3541"
format_stamp: "Formatted at 2026-09-26 07:36:20 on dist-test-slave-qlq8"
server_key: "ddd07c89a01721b410698930bae545a6"
server_key_iv: "cac9ca97204c8d4f7306f2a124e42804"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:20.185316 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:36:20.199855 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:20.201263 3634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:20.201443 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "a398eb0140ae46e88f500999e5ea3541"
format_stamp: "Formatted at 2026-09-26 07:36:20 on dist-test-slave-qlq8"
server_key: "ddd07c89a01721b410698930bae545a6"
server_key_iv: "cac9ca97204c8d4f7306f2a124e42804"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:20.201671 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:20.231335 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:20.234957 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:20.235313 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:20.236224 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:20.237520 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:20.237624 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:20.237768 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:20.237845 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:20.270434 3634 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:41429
I20260926 07:36:20.270471 3761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:41429 every 8 connection(s)
I20260926 07:36:20.271703 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:36:20.277175 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 3634
I20260926 07:36:20.277356 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:36:20.277832 23413 external_mini_cluster.cc:1428] Setting key f7fa56a38a3d0b9e3a43a31a90cf6f8c
Sep 26 07:36:20 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408180, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:20.290982 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:20.274231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:42855 (local address 127.22.221.126:39209)
0926 07:36:20.274706 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:20.274710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:20.275687 (+ 977us) server_negotiation.cc:407] Connection header received
0926 07:36:20.277462 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:20.277468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:20.277563 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:20.277684 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:20.279306 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.280259 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.282023 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.282574 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.282669 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:20.285867 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:20.285871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:20.285874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:20.285906 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:20.288370 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.289001 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.289005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.289007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.289182 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.289561 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.289564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.289566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.289816 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:20.289923 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:20.290585 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 07:36:20.290746 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":217,"threads_started":1}
I20260926 07:36:20.292249 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:20.274833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39209 (local address 127.22.221.67:42855)
0926 07:36:20.275537 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:20.275574 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:20.277169 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:20.277914 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:20.277947 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:20.278443 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:20.279093 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:20.279119 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.280421 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.280427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:20.281892 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:20.281904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.282758 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.282763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:20.282962 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:20.283658 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:20.283680 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:20.285665 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:20.288488 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:20.288494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:20.288506 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:20.288904 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:20.289277 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:20.289281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:20.289283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:20.289464 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:20.289917 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:20.289922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:20.290275 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 07:36:20.290545 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 07:36:20.290946 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":265,"threads_started":1}
I20260926 07:36:20.293661 3762 heartbeater.cc:347] Connected to a master server at 127.22.221.126:39209
I20260926 07:36:20.293907 3762 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:20.294516 3762 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:20.295881 3271 ts_manager.cc:194] Registered new tserver with Master: a398eb0140ae46e88f500999e5ea3541 (127.22.221.67:41429)
I20260926 07:36:20.296720 3271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:42855
I20260926 07:36:20.305696 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:36:20.307518 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 3352
Sep 26 07:36:20 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:36:20.323876 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:20.308355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.22.221.65:46709)
0926 07:36:20.308942 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:20.308950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:20.308975 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:36:20.309036 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:20.309042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:20.309255 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:20.309385 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:20.310831 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.311943 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.313464 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.314005 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.314100 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:20.318472 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:20.318483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:20.318510 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:20.318541 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:20.321377 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.322009 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.322015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.322018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.322140 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.322462 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.322465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.322467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.322778 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:20.322998 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:20.323277 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 07:36:20.323476 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
I20260926 07:36:20.325052 3480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:20.406248 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 3352
I20260926 07:36:20.422289 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 3491
Sep 26 07:36:20 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:36:20.438038 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:20.423176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.22.221.66:40967)
0926 07:36:20.423776 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:20.423783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:20.423808 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:36:20.423869 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:20.423876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:20.424095 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:20.424223 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:20.425466 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.426644 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.428016 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.428530 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.428624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:20.432722 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:20.432737 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:20.432773 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:20.432832 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:20.435644 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.436273 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.436279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.436282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.436404 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.436761 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.436764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.436766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.437078 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:20.437302 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:20.437562 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 07:36:20.437742 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":340,"threads_started":1}
I20260926 07:36:20.439349 3623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:20.526111 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 3491
I20260926 07:36:20.542140 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 3634
Sep 26 07:36:20 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:36:20.558066 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:20.543182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.22.221.67:41429)
0926 07:36:20.543881 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:20.543887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:20.543915 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:36:20.543973 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:20.543979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:20.544199 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:20.544331 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:20.545712 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.546727 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.548073 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:20.548604 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:20.548702 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:20.552927 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:20.552936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:20.552951 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:20.552986 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:20.555790 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.556394 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.556400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.556402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.556542 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:20.556838 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:20.556841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:20.556842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:20.557136 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:20.557357 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:20.557615 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 07:36:20.557798 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":355,"threads_started":1}
I20260926 07:36:20.559361 3757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:20.642283 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 3634
I20260926 07:36:20.658051 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 3240
I20260926 07:36:20.659343 3302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:20.765694 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 3240
I20260926 07:36:20.787223 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39209
--webserver_interface=127.22.221.126
--webserver_port=34405
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:39209
--ranger_config_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:21.111930 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:21.112293 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:21.112460 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:21.119756 3774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:21.119925 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:21.120038 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:21.120141 3774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:21.120230 3774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:21.130595 3774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:39209
--ranger_config_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=34405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:21.132052 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:21.133407 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:21.145265 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:21.145265 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:21.145318 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:21.145999 3774 server_base.cc:1034] running on GCE node
I20260926 07:36:21.146662 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:21.148124 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:21.149361 3774 hybrid_clock.cc:648] HybridClock initialized: now 1790408181149324 us; error 25 us; skew 500 ppm
I20260926 07:36:21.149724 3774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:21 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408181, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:21.152962 3774 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:21.154331 3774 webserver.cc:460] Webserver started at http://127.22.221.126:34405/ using document root <none> and password file <none>
I20260926 07:36:21.154980 3774 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:21.155129 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:36:21 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408181, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:21.210013 3774 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:36:21.213080 3790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:21.214480 3774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:36:21.214684 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd"
format_stamp: "Formatted at 2026-09-26 07:36:17 on dist-test-slave-qlq8"
server_key: "892ca1172cce6c2efd5bf2ae5245ef94"
server_key_iv: "f255af75969383a4b02b2814c3b9ac6e"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:21.215723 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:21.245518 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:21.249323 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:21.249621 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:21.267771 3774 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:39209
I20260926 07:36:21.267789 3841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:39209 every 8 connection(s)
I20260926 07:36:21.269335 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:21.270797 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 3774
I20260926 07:36:21.271934 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:46709
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:39209
--webserver_port=43131
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:21.279337 3842 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:36:21.282994 3842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Bootstrap starting.
I20260926 07:36:21.302239 3842 log.cc:824] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:21.310866 3842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:36:21.311461 3842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Bootstrap complete.
I20260926 07:36:21.320410 3842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:21.321235 3842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb86cf4f3d74649ae4e57840ba6ebcd, State: Initialized, Role: FOLLOWER
I20260926 07:36:21.321841 3842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:21.322081 3842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:21.322194 3842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:21.322384 3842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:36:21.326030 3842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:21.326514 3842 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb86cf4f3d74649ae4e57840ba6ebcd; no voters:
I20260926 07:36:21.327139 3842 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:36:21.327289 3846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:36:21.328656 3846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [term 2 LEADER]: Becoming Leader. State: Replica: 6fb86cf4f3d74649ae4e57840ba6ebcd, State: Running, Role: LEADER
I20260926 07:36:21.329174 3846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } }
I20260926 07:36:21.329469 3842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:21.335497 3847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } } }
I20260926 07:36:21.335976 3847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:21.335896 3848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb86cf4f3d74649ae4e57840ba6ebcd. Latest consensus state: current_term: 2 leader_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb86cf4f3d74649ae4e57840ba6ebcd" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39209 } } }
I20260926 07:36:21.336233 3848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:21.348079 3857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:21.353611 3857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:21.354447 3842 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:36:21.355084 3857 catalog_manager.cc:1259] Loaded cluster ID: ca74d5602f1342518624290c5c1985bb
I20260926 07:36:21.355170 3857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:21.359400 3857 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:21.361085 3857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Loaded TSK: 0
I20260926 07:36:21.362627 3857 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:36:21.718703 3844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:21.719153 3844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:21.719381 3844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:21.732024 3844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:21.732163 3844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:21.732368 3844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:36:21.748894 3844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:46709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=43131
--tserver_master_addrs=127.22.221.126:39209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:21.750541 3844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:21.752334 3844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:21.794847 3880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:21.798336 3881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:21.800122 3883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:21.807642 3844 server_base.cc:1034] running on GCE node
I20260926 07:36:21.808537 3844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:21.819757 3844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:21.821130 3844 hybrid_clock.cc:648] HybridClock initialized: now 1790408181821008 us; error 127 us; skew 500 ppm
I20260926 07:36:21.821596 3844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:21 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408181, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:21.826033 3844 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:36:21.859731 3844 webserver.cc:460] Webserver started at http://127.22.221.65:43131/ using document root <none> and password file <none>
I20260926 07:36:21.861234 3844 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:21.861519 3844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:36:21 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408181, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:21.913812 3844 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.000s
I20260926 07:36:21.918079 3892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:21.920130 3844 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:36:21.920359 3844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "1b35bc6bfc224dcab618dec4e0c76c67"
format_stamp: "Formatted at 2026-09-26 07:36:18 on dist-test-slave-qlq8"
server_key: "06525315d45c2ceb839700e35e58c644"
server_key_iv: "9afe4248ebce17fee489cb00f4520493"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:21.921448 3844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:21.952558 3844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:21.958317 3844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:21.958655 3844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:21.960078 3844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:21.961768 3844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:21.961901 3844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:21.962049 3844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:21.962137 3844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:22.028963 3844 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:46709
I20260926 07:36:22.029057 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:46709 every 8 connection(s)
I20260926 07:36:22.031468 3844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:36:22.039477 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 3844
I20260926 07:36:22.040320 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:40967
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:39209
--webserver_port=45345
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:36:22 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408181, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:22.104877 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:22.056144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:57415 (local address 127.22.221.126:39209)
0926 07:36:22.057230 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:22.057253 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:22.069652 (+ 12399us) server_negotiation.cc:407] Connection header received
0926 07:36:22.074241 (+ 4589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:22.074270 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:22.074905 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:22.075485 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:22.078559 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.080893 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.083753 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.084801 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.084956 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:22.095183 (+ 10227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:22.095194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:22.095214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:22.095305 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:22.098999 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:22.100076 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:22.100086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:22.100094 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:22.100297 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:22.100925 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:22.100931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:22.100935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:22.101610 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:22.101791 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:22.102918 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0926 07:36:22.103380 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":323,"threads_started":1}
I20260926 07:36:22.107177 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:22.054852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39209 (local address 127.22.221.65:57415)
0926 07:36:22.069349 (+ 14497us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:22.069437 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:22.073353 (+ 3916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:22.076329 (+ 2976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:22.076345 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:22.077233 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:22.078229 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:22.078262 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.081186 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.081194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:22.083546 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:22.083577 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.085395 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.085402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:22.085677 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:22.091893 (+ 6216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:22.091934 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:22.094818 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:22.099187 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:22.099197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:22.099213 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:22.099880 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:22.100486 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:22.100491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:22.100510 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:22.100779 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:22.101934 (+ 1155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:22.101941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:22.102341 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 07:36:22.105455 (+ 3114us) client_negotiation.cc:240] Negotiation successful
0926 07:36:22.105998 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14061,"thread_start_us":16263,"threads_started":1}
I20260926 07:36:22.110962 4005 heartbeater.cc:347] Connected to a master server at 127.22.221.126:39209
I20260926 07:36:22.111320 4005 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:22.112192 4005 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
W20260926 07:36:22.115406 4005 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:39209 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:36:22.115629 4005 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:22.115947 4005 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:22.117352 4005 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:22.117681 4005 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
W20260926 07:36:22.118757 4005 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:36:22.485987 4009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:22.486455 4009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:22.486644 4009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:22.498325 4009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:22.498550 4009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:22.498777 4009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:36:22.514598 4009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:40967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=45345
--tserver_master_addrs=127.22.221.126:39209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:22.516285 4009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:22.517922 4009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:22.535548 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:22.535133 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:22.544786 4009 server_base.cc:1034] running on GCE node
W20260926 07:36:22.545009 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:22.545895 4009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:22.547197 4009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:22.548544 4009 hybrid_clock.cc:648] HybridClock initialized: now 1790408182548473 us; error 69 us; skew 500 ppm
I20260926 07:36:22.548947 4009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:22 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:22.553113 4009 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
Sep 26 07:36:22 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790408182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:22.562157 4009 webserver.cc:460] Webserver started at http://127.22.221.66:45345/ using document root <none> and password file <none>
I20260926 07:36:22.563201 4009 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:22.563436 4009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:36:22 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:22.599159 4009 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.004s
I20260926 07:36:22.603060 4036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:22.604861 4009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:36:22.605044 4009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "0323636d511641b9aa3423bafd13e895"
format_stamp: "Formatted at 2026-09-26 07:36:19 on dist-test-slave-qlq8"
server_key: "94c136ceb16f7a431f9e5f2abad18e0e"
server_key_iv: "5e3a33b006cc406cb862106332d9282d"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:22.606478 4009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:22.629513 4009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:22.635105 4009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:22.635447 4009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:22.636651 4009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:22.638428 4009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:22.638541 4009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:22.638650 4009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:22.638727 4009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:22.695432 4009 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:40967
I20260926 07:36:22.695477 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:40967 every 8 connection(s)
I20260926 07:36:22.697038 4009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:36:22.704677 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4009
I20260926 07:36:22.705477 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.221.84:34541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:41429
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:39209
--webserver_port=37107
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:36:22 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:22.733310 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:22.700912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39209 (local address 127.22.221.66:32849)
0926 07:36:22.701832 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:22.701897 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:22.704500 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:22.705678 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:22.705692 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:22.706307 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:22.707144 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:22.707172 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.708747 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.708752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:22.710853 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:22.710879 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.712191 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.712198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:22.712512 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:22.713340 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:22.713372 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:22.719706 (+ 6334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:22.728280 (+ 8574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:22.728290 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:22.728308 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:22.728929 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:22.729427 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:22.729433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:22.729437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:22.729696 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:22.730419 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:22.730427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:22.730912 (+ 485us) client_negotiation.cc:769] Sending connection context
0926 07:36:22.731253 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 07:36:22.731877 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":382,"threads_started":1}
I20260926 07:36:22.734552 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:22.700199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:32849 (local address 127.22.221.126:39209)
0926 07:36:22.700800 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:22.700807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:22.702033 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 07:36:22.704879 (+ 2846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:22.704887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:22.704987 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:22.705076 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:22.707394 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.708527 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.711047 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:22.711678 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:22.711804 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:22.720016 (+ 8212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:22.720023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:22.720029 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:22.720075 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:22.727177 (+ 7102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:22.729099 (+ 1922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:22.729105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:22.729109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:22.729257 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:22.729835 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:22.729840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:22.729843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:22.730200 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:22.730339 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:22.733887 (+ 3548us) server_negotiation.cc:299] Negotiation successful
0926 07:36:22.734144 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":322,"threads_started":1}
I20260926 07:36:22.736866 4150 heartbeater.cc:347] Connected to a master server at 127.22.221.126:39209
I20260926 07:36:22.737216 4150 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:22.738140 4150 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:22.740342 3807 ts_manager.cc:194] Registered new tserver with Master: 0323636d511641b9aa3423bafd13e895 (127.22.221.66:40967)
I20260926 07:36:22.742728 3807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:32849
Sep 26 07:36:22 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:23.118918 4005 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:23.119444 4005 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:23.120767 3807 ts_manager.cc:194] Registered new tserver with Master: 1b35bc6bfc224dcab618dec4e0c76c67 (127.22.221.65:46709)
I20260926 07:36:23.121961 3807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:57415
W20260926 07:36:23.186259 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:23.186626 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:23.186766 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:23.197883 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:23.198045 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:23.198221 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:36:23.214079 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.221.84:34541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:41429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=37107
--tserver_master_addrs=127.22.221.126:39209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:23.215829 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:23.217553 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:23.252987 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:23.255641 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:23.267482 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:23.279639 4154 server_base.cc:1034] running on GCE node
I20260926 07:36:23.281425 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:23.289819 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:23.299475 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790408183299360 us; error 124 us; skew 500 ppm
I20260926 07:36:23.300012 4154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:23 dist-test-slave-qlq8 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408183, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:23.304687 4154 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:36:23.329908 4154 webserver.cc:460] Webserver started at http://127.22.221.67:37107/ using document root <none> and password file <none>
I20260926 07:36:23.331107 4154 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:23.331378 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:36:23 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408182, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
Sep 26 07:36:23 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408183, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for HTTP/127.22.221.84@KRBTEST.COM
I20260926 07:36:23.385180 4154 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.004s sys 0.000s
I20260926 07:36:23.396028 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:23.404619 4154 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.005s
I20260926 07:36:23.404865 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "a398eb0140ae46e88f500999e5ea3541"
format_stamp: "Formatted at 2026-09-26 07:36:20 on dist-test-slave-qlq8"
server_key: "ddd07c89a01721b410698930bae545a6"
server_key_iv: "cac9ca97204c8d4f7306f2a124e42804"
server_key_version: "kuduclusterkey@0"
I20260926 07:36:23.406334 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:23.505100 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:23.510432 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:23.510797 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:23.518249 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:23.526343 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:23.526543 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:23.526705 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:23.526826 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:23.691572 4154 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:41429
I20260926 07:36:23.691610 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:41429 every 8 connection(s)
I20260926 07:36:23.693408 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:36:23.697166 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4154
Sep 26 07:36:23 dist-test-slave-qlq8 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408183, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:23.715083 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.698628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.22.221.126:39209)
0926 07:36:23.699425 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.699430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.699449 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:36:23.699511 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.699515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.699608 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.699726 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:23.701767 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.702697 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.704499 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.704914 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.705001 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.708061 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:23.708068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:23.708073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:23.708116 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:23.711617 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:23.712706 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:23.712710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:23.712713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:23.712828 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:23.713302 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:23.713308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:23.713311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:23.713681 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:23.713982 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.714392 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.714637 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":507,"threads_started":1}
I20260926 07:36:23.717223 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.696350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:48265 (local address 127.22.221.126:39209)
0926 07:36:23.696883 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.696898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.697975 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 07:36:23.700432 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.700437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.700516 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.700617 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:23.702372 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.703355 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.705367 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.705783 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.705876 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.709404 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:23.709409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:23.709412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:23.709475 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:23.713024 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:23.713975 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:23.713981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:23.713985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:23.714148 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:23.714713 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:23.714718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:23.714722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:23.715076 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:23.715244 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.716551 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.716778 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":294,"threads_started":1}
I20260926 07:36:23.718132 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.696933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:39209 (local address 127.22.221.67:48265)
0926 07:36:23.697805 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:23.697856 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:23.700135 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:23.700906 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:23.700919 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:23.701520 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:23.702156 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.702172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.703505 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.703511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.705209 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.705234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.706001 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.706007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.706250 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.706878 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:23.706906 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:23.709135 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:23.713170 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:23.713178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:23.713192 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:23.713835 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:23.714294 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:23.714298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:23.714302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:23.714569 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:23.715584 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:23.715593 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:23.716043 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 07:36:23.716393 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 07:36:23.716949 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":338,"threads_started":1}
I20260926 07:36:23.719522 4291 heartbeater.cc:347] Connected to a master server at 127.22.221.126:39209
I20260926 07:36:23.719812 4291 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:23.720590 4291 heartbeater.cc:511] Master 127.22.221.126:39209 requested a full tablet report, sending...
I20260926 07:36:23.721946 3807 ts_manager.cc:194] Registered new tserver with Master: a398eb0140ae46e88f500999e5ea3541 (127.22.221.67:41429)
I20260926 07:36:23.722739 3807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:48265
I20260926 07:36:23.730633 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:36:23.747058 4150 heartbeater.cc:503] Master 127.22.221.126:39209 was elected leader, sending a full tablet report...
I20260926 07:36:23.748027 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.733859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.22.221.126:39209)
0926 07:36:23.734024 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.734029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.734333 (+ 304us) server_negotiation.cc:407] Connection header received
0926 07:36:23.734519 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.734523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.734589 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.734673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:23.736409 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.737167 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.738467 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.738865 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.738969 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.741629 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:23.741633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:23.741637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:23.741667 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:23.744276 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:23.745068 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:23.745072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:23.745074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:23.745212 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:23.745620 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:23.745628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:23.745631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:23.746186 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:23.746382 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.746809 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.747031 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:36:23.753571 3806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:36:23.761415 3806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:36:23.796137 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.775977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.22.221.66:40967)
0926 07:36:23.787799 (+ 11822us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.787811 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.788639 (+ 828us) server_negotiation.cc:407] Connection header received
0926 07:36:23.788757 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.788767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.789050 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.789261 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:23.790033 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.791015 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.793148 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.794685 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.794845 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.795065 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.795581 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.795740 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11622,"thread_start_us":11448,"threads_started":1}
I20260926 07:36:23.796072 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.775877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:40967 (local address 127.0.0.1:53562)
0926 07:36:23.788237 (+ 12360us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:23.788286 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:23.788498 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:23.789317 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:23.789325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:23.789359 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:23.789842 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.789852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.791165 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.791171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.792963 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.792979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.794919 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.794927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.795147 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.795208 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 07:36:23.795384 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 07:36:23.795670 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12080,"thread_start_us":11947,"threads_started":1}
I20260926 07:36:23.801108 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.776341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.22.221.65:46709)
0926 07:36:23.789123 (+ 12782us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.789130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.792754 (+ 3624us) server_negotiation.cc:407] Connection header received
0926 07:36:23.792911 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.792919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.793167 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.793369 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:23.794276 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.795828 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.798589 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.800033 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.800184 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.800424 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.800624 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.800780 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12610,"thread_start_us":12463,"threads_started":1}
I20260926 07:36:23.801154 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.775882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48564)
0926 07:36:23.792605 (+ 16723us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:23.792625 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:23.792747 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:23.793598 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:23.793604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:23.793631 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:23.794021 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.794034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.795998 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.796003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.798396 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.798413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.800252 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.800258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.800423 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.800463 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:36:23.800582 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:36:23.800707 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16588,"mutex_wait_us":649,"thread_start_us":17558,"threads_started":1}
I20260926 07:36:23.801088 4085 tablet_service.cc:1467] Processing CreateTablet for tablet e1342df992064d05867d683dfd0ebf63 (DEFAULT_TABLE table=test-table [id=70cf1b5ba3ef45779f31c204905836d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:23.802807 4085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1342df992064d05867d683dfd0ebf63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:23.803581 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.777441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:41429 (local address 127.0.0.1:59336)
0926 07:36:23.794370 (+ 16929us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:23.794384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:23.794537 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:23.795189 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:23.795194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:23.795220 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:23.795706 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.795719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.797654 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.797658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.799747 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.799763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.802687 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.802693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.802872 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.802915 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:36:23.803025 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:36:23.803155 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16829,"mutex_wait_us":61,"thread_start_us":14887,"threads_started":1}
I20260926 07:36:23.803608 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.778154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.22.221.67:41429)
0926 07:36:23.793329 (+ 15175us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.793337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.794496 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 07:36:23.794699 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.794705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.794941 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.795137 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:23.795909 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.797510 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.799933 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.802451 (+ 2518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.802592 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.802815 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.803083 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.803264 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14995,"thread_start_us":15503,"threads_started":1}
I20260926 07:36:23.805737 3940 tablet_service.cc:1467] Processing CreateTablet for tablet e1342df992064d05867d683dfd0ebf63 (DEFAULT_TABLE table=test-table [id=70cf1b5ba3ef45779f31c204905836d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:23.807716 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1342df992064d05867d683dfd0ebf63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:23.809386 4219 tablet_service.cc:1467] Processing CreateTablet for tablet e1342df992064d05867d683dfd0ebf63 (DEFAULT_TABLE table=test-table [id=70cf1b5ba3ef45779f31c204905836d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:23.810850 4219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1342df992064d05867d683dfd0ebf63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:23.816761 4312 tablet_bootstrap.cc:492] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Bootstrap starting.
I20260926 07:36:23.820729 4312 tablet_bootstrap.cc:654] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:23.821777 4312 log.cc:824] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:23.824862 4314 tablet_bootstrap.cc:492] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: Bootstrap starting.
I20260926 07:36:23.825029 4312 tablet_bootstrap.cc:492] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: No bootstrap required, opened a new log
I20260926 07:36:23.825353 4312 ts_tablet_manager.cc:1397] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 07:36:23.826445 4315 tablet_bootstrap.cc:492] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Bootstrap starting.
I20260926 07:36:23.828634 4314 tablet_bootstrap.cc:654] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:23.829957 4314 log.cc:824] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:23.830360 4315 tablet_bootstrap.cc:654] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:23.831671 4315 log.cc:824] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:23.833182 4314 tablet_bootstrap.cc:492] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: No bootstrap required, opened a new log
I20260926 07:36:23.833489 4314 ts_tablet_manager.cc:1397] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 07:36:23.834444 4315 tablet_bootstrap.cc:492] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: No bootstrap required, opened a new log
I20260926 07:36:23.834739 4315 ts_tablet_manager.cc:1397] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 07:36:23.837226 4312 raft_consensus.cc:348] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.837679 4312 raft_consensus.cc:374] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:23.837821 4312 raft_consensus.cc:729] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0323636d511641b9aa3423bafd13e895, State: Initialized, Role: FOLLOWER
I20260926 07:36:23.838423 4312 consensus_queue.cc:260] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.841645 4312 ts_tablet_manager.cc:1428] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 07:36:23.843178 4315 raft_consensus.cc:348] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.843592 4315 raft_consensus.cc:374] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:23.843719 4315 raft_consensus.cc:729] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a398eb0140ae46e88f500999e5ea3541, State: Initialized, Role: FOLLOWER
I20260926 07:36:23.844291 4315 consensus_queue.cc:260] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.844244 4314 raft_consensus.cc:348] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.844666 4314 raft_consensus.cc:374] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:23.844807 4314 raft_consensus.cc:729] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b35bc6bfc224dcab618dec4e0c76c67, State: Initialized, Role: FOLLOWER
I20260926 07:36:23.845399 4291 heartbeater.cc:503] Master 127.22.221.126:39209 was elected leader, sending a full tablet report...
I20260926 07:36:23.845422 4314 consensus_queue.cc:260] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.846009 4315 ts_tablet_manager.cc:1428] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Time spent starting tablet: real 0.011s user 0.011s sys 0.002s
I20260926 07:36:23.846819 4005 heartbeater.cc:503] Master 127.22.221.126:39209 was elected leader, sending a full tablet report...
I20260926 07:36:23.847506 4314 ts_tablet_manager.cc:1428] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260926 07:36:23.887459 4320 raft_consensus.cc:482] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:36:23.887830 4320 raft_consensus.cc:504] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.890347 4320 leader_election.cc:288] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0323636d511641b9aa3423bafd13e895 (127.22.221.66:40967), a398eb0140ae46e88f500999e5ea3541 (127.22.221.67:41429)
I20260926 07:36:23.897569 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.890574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:34959 (local address 127.22.221.66:40967)
0926 07:36:23.890756 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.890764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.891947 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 07:36:23.892191 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.892198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.892298 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.892391 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:23.893074 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.893816 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.895762 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.896556 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.896673 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.896781 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.897153 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.897303 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:36:23.897598 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.891006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:55999 (local address 127.22.221.67:41429)
0926 07:36:23.891285 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.891294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.892333 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 07:36:23.892445 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.892450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.892525 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.892638 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:23.893238 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.894018 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.895834 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.896588 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.896792 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.896886 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.897221 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.897346 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:36:23.897768 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.891402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:41429 (local address 127.22.221.65:55999)
0926 07:36:23.892191 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:23.892207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:23.892328 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:23.892660 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:23.892663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:23.892685 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:23.893049 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.893059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.894163 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.894166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.895648 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.895659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.896866 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.896871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.897029 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.897051 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:36:23.897167 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:36:23.897294 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":54,"thread_start_us":421,"threads_started":1}
I20260926 07:36:23.897804 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.890513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:40967 (local address 127.22.221.65:34959)
0926 07:36:23.891715 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:23.891754 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:23.892009 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:23.892442 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:23.892447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:23.892487 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:23.892920 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.892934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.893957 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.893963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.895614 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:23.895626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.896767 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.896774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:23.896970 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.897019 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:36:23.897093 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:36:23.897240 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":9,"thread_start_us":816,"threads_started":1}
I20260926 07:36:23.899034 4239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1342df992064d05867d683dfd0ebf63" candidate_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a398eb0140ae46e88f500999e5ea3541" is_pre_election: true
I20260926 07:36:23.899015 4105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1342df992064d05867d683dfd0ebf63" candidate_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0323636d511641b9aa3423bafd13e895" is_pre_election: true
I20260926 07:36:23.899628 4239 raft_consensus.cc:2454] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b35bc6bfc224dcab618dec4e0c76c67 in term 0.
I20260926 07:36:23.899628 4105 raft_consensus.cc:2454] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b35bc6bfc224dcab618dec4e0c76c67 in term 0.
I20260926 07:36:23.900359 3895 leader_election.cc:302] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b35bc6bfc224dcab618dec4e0c76c67, a398eb0140ae46e88f500999e5ea3541; no voters:
I20260926 07:36:23.900859 4320 raft_consensus.cc:2781] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:36:23.900979 4320 raft_consensus.cc:482] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:36:23.901099 4320 raft_consensus.cc:3037] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:23.904409 4320 raft_consensus.cc:504] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.905149 4320 leader_election.cc:288] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [CANDIDATE]: Term 1 election: Requested vote from peers 0323636d511641b9aa3423bafd13e895 (127.22.221.66:40967), a398eb0140ae46e88f500999e5ea3541 (127.22.221.67:41429)
I20260926 07:36:23.905624 4105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1342df992064d05867d683dfd0ebf63" candidate_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0323636d511641b9aa3423bafd13e895"
I20260926 07:36:23.905635 4239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1342df992064d05867d683dfd0ebf63" candidate_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a398eb0140ae46e88f500999e5ea3541"
I20260926 07:36:23.905840 4105 raft_consensus.cc:3037] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:23.905848 4239 raft_consensus.cc:3037] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:23.908092 4105 raft_consensus.cc:2454] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b35bc6bfc224dcab618dec4e0c76c67 in term 1.
I20260926 07:36:23.908154 4239 raft_consensus.cc:2454] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b35bc6bfc224dcab618dec4e0c76c67 in term 1.
I20260926 07:36:23.908530 3893 leader_election.cc:302] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0323636d511641b9aa3423bafd13e895, 1b35bc6bfc224dcab618dec4e0c76c67; no voters:
I20260926 07:36:23.908836 4320 raft_consensus.cc:2781] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:23.909240 4320 raft_consensus.cc:686] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 1 LEADER]: Becoming Leader. State: Replica: 1b35bc6bfc224dcab618dec4e0c76c67, State: Running, Role: LEADER
I20260926 07:36:23.909718 4320 consensus_queue.cc:237] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } }
I20260926 07:36:23.914455 3807 catalog_manager.cc:5495] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b35bc6bfc224dcab618dec4e0c76c67 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b35bc6bfc224dcab618dec4e0c76c67" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 46709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 } health_report { overall_health: UNKNOWN } } }
W20260926 07:36:23.945053 4292 tablet.cc:2369] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:36:23.950407 4151 tablet.cc:2369] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:36:23.956346 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:23.950277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.22.221.65:46709)
0926 07:36:23.950468 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:23.950472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:23.950496 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:36:23.950739 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:23.950742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:23.950831 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:23.950955 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:36:23.951491 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.952333 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.953784 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:23.954379 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:23.954499 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:23.955078 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:36:23.955671 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:36:23.955789 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:23.955949 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:36:23.956020 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:36:23.964632 4239 raft_consensus.cc:1261] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 1 FOLLOWER]: Refusing update from remote peer 1b35bc6bfc224dcab618dec4e0c76c67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:23.964627 4105 raft_consensus.cc:1261] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 1 FOLLOWER]: Refusing update from remote peer 1b35bc6bfc224dcab618dec4e0c76c67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:23.965590 4323 consensus_queue.cc:1035] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0323636d511641b9aa3423bafd13e895" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:23.965965 4320 consensus_queue.cc:1035] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a398eb0140ae46e88f500999e5ea3541" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:23.978058 4327 mvcc.cc:205] Tried to move back new op lower bound from 7333511921509076992 to 7333511921297772544. Current Snapshot: MvccSnapshot[applied={T|T < 7333511921509076992}]
I20260926 07:36:23.990760 3806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:36:23.991066 3806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:36:23.994892 3806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb86cf4f3d74649ae4e57840ba6ebcd: Sending DeleteTablet for 3 replicas of tablet e1342df992064d05867d683dfd0ebf63
I20260926 07:36:23.996037 4085 tablet_service.cc:1514] Processing DeleteTablet for tablet e1342df992064d05867d683dfd0ebf63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:23 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:53562
I20260926 07:36:23.996269 4219 tablet_service.cc:1514] Processing DeleteTablet for tablet e1342df992064d05867d683dfd0ebf63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:23 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:59336
I20260926 07:36:23.999598 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet e1342df992064d05867d683dfd0ebf63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:23 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:48564
I20260926 07:36:24.001331 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 3844
I20260926 07:36:24.006989 4336 tablet_replica.cc:330] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: stopping tablet replica
I20260926 07:36:24.007470 4336 raft_consensus.cc:2229] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:36:24.007968 4336 raft_consensus.cc:2258] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:24.011538 4337 tablet_replica.cc:330] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: stopping tablet replica
I20260926 07:36:24.012077 4337 raft_consensus.cc:2229] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:36:24.012735 4337 raft_consensus.cc:2258] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:24.014894 4336 ts_tablet_manager.cc:1905] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:24.015622 4337 ts_tablet_manager.cc:1905] T e1342df992064d05867d683dfd0ebf63 P 1b35bc6bfc224dcab618dec4e0c76c67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:24.020306 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.002308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.22.221.65:46709)
0926 07:36:24.002464 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:24.002470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:24.002549 (+ 79us) server_negotiation.cc:407] Connection header received
0926 07:36:24.002773 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:24.002781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:24.002884 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:24.002997 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:24.004810 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:24.006262 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:24.008367 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:24.009128 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:24.009264 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:24.012817 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:24.012831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:24.012851 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:24.012900 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:24.017003 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:24.017872 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:24.017880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:24.017883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:24.018098 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:24.018558 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:24.018563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:24.018568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:24.019038 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:24.019417 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:24.019701 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:36:24.019942 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:36:24.021062 4000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:24.022680 4335 tablet_replica.cc:330] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: stopping tablet replica
I20260926 07:36:24.022644 4336 ts_tablet_manager.cc:1918] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:24.022864 4336 log.cc:1196] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/e1342df992064d05867d683dfd0ebf63
I20260926 07:36:24.023150 4335 raft_consensus.cc:2229] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:36:24.023542 4336 ts_tablet_manager.cc:1939] T e1342df992064d05867d683dfd0ebf63 P a398eb0140ae46e88f500999e5ea3541: Deleting consensus metadata
I20260926 07:36:24.023713 4335 raft_consensus.cc:2258] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:24.026055 4335 ts_tablet_manager.cc:1905] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:24.026118 3793 catalog_manager.cc:4841] TS a398eb0140ae46e88f500999e5ea3541 (127.22.221.67:41429): tablet e1342df992064d05867d683dfd0ebf63 (table test-table [id=70cf1b5ba3ef45779f31c204905836d0]) successfully deleted
I20260926 07:36:24.033686 4335 ts_tablet_manager.cc:1918] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:24.033915 4335 log.cc:1196] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/e1342df992064d05867d683dfd0ebf63
I20260926 07:36:24.034577 4335 ts_tablet_manager.cc:1939] T e1342df992064d05867d683dfd0ebf63 P 0323636d511641b9aa3423bafd13e895: Deleting consensus metadata
I20260926 07:36:24.036815 3791 catalog_manager.cc:4841] TS 0323636d511641b9aa3423bafd13e895 (127.22.221.66:40967): tablet e1342df992064d05867d683dfd0ebf63 (table test-table [id=70cf1b5ba3ef45779f31c204905836d0]) successfully deleted
I20260926 07:36:24.122884 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 3844
I20260926 07:36:24.140759 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4009
W20260926 07:36:24.140707 3794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.221.65:46709 (error 108)
I20260926 07:36:24.142668 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.142209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48592)
0926 07:36:24.142388 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.142506 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:36:24.142938 3794 catalog_manager.cc:4584] TS 1b35bc6bfc224dcab618dec4e0c76c67 (127.22.221.65:46709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1342df992064d05867d683dfd0ebf63: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
I20260926 07:36:24.160903 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.141875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.22.221.66:40967)
0926 07:36:24.142085 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:24.142089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:24.142106 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:36:24.142189 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:24.142196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:24.142298 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:24.142405 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:24.143966 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:24.144721 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:24.146113 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:24.146714 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:24.146858 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:24.154613 (+ 7755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:24.154629 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:24.154646 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:24.154706 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:24.157896 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:24.158618 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:24.158633 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:24.158635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:24.158788 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:24.159188 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:24.159193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:24.159196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:24.159658 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:24.159974 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:24.160303 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 07:36:24.160537 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:36:24.161708 4145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:24.196573 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.196104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48598)
0926 07:36:24.196304 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.196415 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:36:24.197657 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.197297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48600)
0926 07:36:24.197466 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.197541 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 07:36:24.252686 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4009
I20260926 07:36:24.258353 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.257860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48604)
0926 07:36:24.258094 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.258193 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:36:24.259498 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.259092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48620)
0926 07:36:24.259290 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.259370 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:36:24.271695 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4154
I20260926 07:36:24.286234 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.272548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.22.221.67:41429)
0926 07:36:24.272718 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:24.272725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:24.272751 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:36:24.273015 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:24.273022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:24.273088 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:24.273216 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:24.274497 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:24.275293 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:24.276611 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:24.277241 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:24.277369 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:24.279937 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:24.279954 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:24.279984 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:24.280021 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:24.283573 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:24.284233 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:24.284242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:24.284246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:24.284381 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:24.284710 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:24.284714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:24.284718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:24.285131 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:24.285418 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:24.285621 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 07:36:24.285830 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:36:24.287010 4281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:24.340826 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.340370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48626)
0926 07:36:24.340580 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.340677 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 07:36:24.341920 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.341547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:48630)
0926 07:36:24.341749 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.341816 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 07:36:24.381522 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4154
I20260926 07:36:24.397928 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 3774
I20260926 07:36:24.399142 3837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:24.518811 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:24.518192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:46709 (local address 127.0.0.1:50014)
0926 07:36:24.518427 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:24.518584 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:36:24.518862 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 3774
I20260926 07:36:24.539645 23413 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:36:24.874583 23413 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:36:24.874790 23413 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:36:25.220180 23413 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:36:25.220 UTC [2826] LOG: received smart shutdown request
2026-09-26 07:36:25.223 UTC [2826] LOG: background worker "logical replication launcher" (PID 2835) exited with exit code 1
2026-09-26 07:36:25.224 UTC [2830] LOG: shutting down
2026-09-26 07:36:25.402 UTC [2826] LOG: database system is shut down
2026-09-26T07:36:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53124 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:36:25 dist-test-slave-qlq8 krb5kdc[4351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:36:25 dist-test-slave-qlq8 krb5kdc[4351](info): set up 2 sockets
Sep 26 07:36:25 dist-test-slave-qlq8 krb5kdc[4351](info): commencing operation
krb5kdc: starting...
W20260926 07:36:26.691917 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:36:26 dist-test-slave-qlq8 krb5kdc[4351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:36:26.927907 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43533
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43533,127.22.221.125:40505,127.22.221.124:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:27.265010 4364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:27.265403 4364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:27.265587 4364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:27.272995 4364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:27.273175 4364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:27.273286 4364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:27.273402 4364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:27.273496 4364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:27.283703 4364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43533,127.22.221.125:40505,127.22.221.124:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:27.285132 4364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:27.286458 4364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:27.298619 4369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:27.298655 4370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:27.298581 4372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:27.299574 4364 server_base.cc:1034] running on GCE node
I20260926 07:36:27.300318 4364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:27.301676 4364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:27.302902 4364 hybrid_clock.cc:648] HybridClock initialized: now 1790408187302876 us; error 50 us; skew 500 ppm
I20260926 07:36:27.303375 4364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:27 dist-test-slave-qlq8 krb5kdc[4351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:27.310463 4364 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:27.312234 4364 webserver.cc:460] Webserver started at http://127.22.221.126:42971/ using document root <none> and password file <none>
I20260926 07:36:27.312930 4364 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:27.313090 4364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:27.313398 4364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:27.316560 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "4e5ec601be374c5884ce4a3e013dd089"
format_stamp: "Formatted at 2026-09-26 07:36:27 on dist-test-slave-qlq8"
server_key: "83256aacafe9e12c63b223dd2f0074ad"
server_key_iv: "eee0bf8178e20f8e0e633067cbe79110"
server_key_version: "encryptionkey@0"
I20260926 07:36:27.317375 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "4e5ec601be374c5884ce4a3e013dd089"
format_stamp: "Formatted at 2026-09-26 07:36:27 on dist-test-slave-qlq8"
server_key: "83256aacafe9e12c63b223dd2f0074ad"
server_key_iv: "eee0bf8178e20f8e0e633067cbe79110"
server_key_version: "encryptionkey@0"
I20260926 07:36:27.322495 4364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:36:27.325977 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:27.327205 4364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:36:27.327452 4364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "4e5ec601be374c5884ce4a3e013dd089"
format_stamp: "Formatted at 2026-09-26 07:36:27 on dist-test-slave-qlq8"
server_key: "83256aacafe9e12c63b223dd2f0074ad"
server_key_iv: "eee0bf8178e20f8e0e633067cbe79110"
server_key_version: "encryptionkey@0"
I20260926 07:36:27.327659 4364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:27.355129 4364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:27.359982 4364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:27.360292 4364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:27.380177 4364 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:43533
I20260926 07:36:27.380235 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:43533 every 8 connection(s)
I20260926 07:36:27.381600 4364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:27.385561 4431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:27.390455 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4364
I20260926 07:36:27.390686 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:36:27.391374 23413 external_mini_cluster.cc:1428] Setting key a90f408685c3cb06499809f7052a5e87
I20260926 07:36:27.392874 4431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:36:27 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.125@KRBTEST.COM" created.
I20260926 07:36:27.417846 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.396046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.22.221.126:43533)
0926 07:36:27.397817 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:27.397828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:27.397879 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:36:27.399828 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:27.399838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:27.400121 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:27.400321 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:27.401497 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.403500 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.404994 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.405948 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.406134 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:27.409777 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:27.409791 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:27.409809 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:27.409856 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:27.413756 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:27.414414 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:27.414420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:27.414422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:27.414571 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:27.415035 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:27.415039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:27.415041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:27.415639 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:27.415822 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:27.416379 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 07:36:27.416783 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1567,"spinlock_wait_cycles":71552,"thread_start_us":1365,"threads_started":1}
I20260926 07:36:27.417884 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.396047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43533 (local address 127.0.0.1:38724)
0926 07:36:27.397189 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.397251 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:27.399468 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:27.400368 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:27.400373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:27.400948 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:27.401288 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:27.401306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.403658 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.403662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:27.404843 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:27.404856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.406161 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.406165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:27.406278 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:27.406954 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:27.406976 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:27.409530 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:27.413879 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:27.413888 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:27.413897 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:27.414271 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:27.414669 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:27.414673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:27.414675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:27.414921 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:27.415793 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:27.415800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:27.415973 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 07:36:27.416185 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 07:36:27.417021 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":162,"thread_start_us":517,"threads_started":1}
I20260926 07:36:27.420807 4431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:36:27.421921 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.421422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60710)
0926 07:36:27.421627 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.421757 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 07:36:27.422164 4380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
I20260926 07:36:27.423725 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.423388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60724)
0926 07:36:27.423548 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.423609 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 07:36:27.423995 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:40505: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab.
Entry for principal kudu/127.22.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab.
I20260926 07:36:27.446832 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:40505
--webserver_interface=127.22.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab
--principal=kudu/127.22.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43533,127.22.221.125:40505,127.22.221.124:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:27.473306 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } attempt: 1
I20260926 07:36:27.474578 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.473997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60730)
0926 07:36:27.474235 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.474384 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:36:27.475770 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.475343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60746)
0926 07:36:27.475503 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.475611 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:36:27.476025 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:40505: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
I20260926 07:36:27.544358 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } attempt: 2
I20260926 07:36:27.545679 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.545199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60762)
0926 07:36:27.545460 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.545552 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 07:36:27.546717 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.546350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60768)
0926 07:36:27.546530 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.546596 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:36:27.546890 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:40505: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
I20260926 07:36:27.638140 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } attempt: 3
I20260926 07:36:27.639508 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.638940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60772)
0926 07:36:27.639195 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.639334 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 07:36:27.640650 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.640228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60780)
0926 07:36:27.640386 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.640479 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 07:36:27.640867 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:40505: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
W20260926 07:36:27.780812 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:27.781093 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:27.781265 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:36:27.784116 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } attempt: 4
I20260926 07:36:27.785526 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.784882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60782)
0926 07:36:27.785199 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.785329 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260926 07:36:27.786758 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.786298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60788)
0926 07:36:27.786500 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.786598 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260926 07:36:27.787019 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:40505: Network error: Client connection negotiation failed: client connection to 127.22.221.125:40505: connect: Connection refused (error 111)
W20260926 07:36:27.792553 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:27.792721 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:27.792827 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:27.792932 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:27.793010 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:27.811997 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43533,127.22.221.125:40505,127.22.221.124:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab
--principal=kudu/127.22.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:27.813277 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:27.814504 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:27.826485 4444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:27.826534 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:27.826570 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:27.827551 4438 server_base.cc:1034] running on GCE node
I20260926 07:36:27.828238 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:27.829526 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:27.830724 4438 hybrid_clock.cc:648] HybridClock initialized: now 1790408187830689 us; error 34 us; skew 500 ppm
I20260926 07:36:27.831023 4438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:27 dist-test-slave-qlq8 krb5kdc[4351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:27.834113 4438 init.cc:377] Logged in from keytab as kudu/127.22.221.125@KRBTEST.COM (short username kudu)
I20260926 07:36:27.835505 4438 webserver.cc:460] Webserver started at http://127.22.221.125:42949/ using document root <none> and password file <none>
I20260926 07:36:27.836138 4438 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:27.836231 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:27.836476 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:27.839432 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data/instance:
uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292"
format_stamp: "Formatted at 2026-09-26 07:36:27 on dist-test-slave-qlq8"
server_key: "7a02108133835cdd30b56fd95050c5a6"
server_key_iv: "65e2fc07f10afe6b15d38bafb6e96ff1"
server_key_version: "encryptionkey@0"
I20260926 07:36:27.840190 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal/instance:
uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292"
format_stamp: "Formatted at 2026-09-26 07:36:27 on dist-test-slave-qlq8"
server_key: "7a02108133835cdd30b56fd95050c5a6"
server_key_iv: "65e2fc07f10afe6b15d38bafb6e96ff1"
server_key_version: "encryptionkey@0"
I20260926 07:36:27.845113 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:36:27.848660 4453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:27.850172 4438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:36:27.850342 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal
uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292"
format_stamp: "Formatted at 2026-09-26 07:36:27 on dist-test-slave-qlq8"
server_key: "7a02108133835cdd30b56fd95050c5a6"
server_key_iv: "65e2fc07f10afe6b15d38bafb6e96ff1"
server_key_version: "encryptionkey@0"
I20260926 07:36:27.850539 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:27.875174 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:27.880412 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:27.880676 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:27.900664 4438 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.125:40505
I20260926 07:36:27.900693 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.125:40505 every 8 connection(s)
I20260926 07:36:27.902107 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
I20260926 07:36:27.905803 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:27.908798 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4438
I20260926 07:36:27.909003 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-1/wal/instance
I20260926 07:36:27.909539 23413 external_mini_cluster.cc:1428] Setting key 50283aab19a976f71a9f45f37a7aef8c
I20260926 07:36:27.915184 4505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:36:27 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.124@KRBTEST.COM" created.
I20260926 07:36:27.939057 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.919468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.22.221.126:43533)
0926 07:36:27.921234 (+ 1766us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:27.921243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:27.921757 (+ 514us) server_negotiation.cc:407] Connection header received
0926 07:36:27.923865 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:27.923871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:27.923981 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:27.924116 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:27.925731 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.926459 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.928179 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.928605 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.928771 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:27.932854 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:27.932859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:27.932863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:27.932908 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:27.935796 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:27.936638 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:27.936646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:27.936651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:27.936773 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:27.937261 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:27.937264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:27.937266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:27.937542 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:27.937718 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:27.938452 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 07:36:27.938672 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":1468,"threads_started":1}
I20260926 07:36:27.940097 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.919548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43533 (local address 127.0.0.1:38732)
0926 07:36:27.921563 (+ 2015us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.921603 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:27.923552 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:27.924379 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:27.924388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:27.924935 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:27.925531 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:27.925549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.926646 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.926651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:27.928006 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:27.928019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.928866 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.928873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:27.929108 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:27.929871 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:27.929898 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:27.932619 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:27.935971 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:27.935981 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:27.936001 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:27.936492 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:27.936893 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:27.936900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:27.936905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:27.937118 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:27.937678 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:27.937688 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:27.938073 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 07:36:27.938357 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 07:36:27.938891 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"thread_start_us":1525,"threads_started":1}
I20260926 07:36:27.942281 4505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:36:27 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:36:27.957347 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.943039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60796)
0926 07:36:27.943214 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.943286 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:27.943421 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:27.944327 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:27.944331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:27.944794 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:27.944940 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:27.944949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.946287 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.946295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:27.947149 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:27.947159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.948051 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.948058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:27.948290 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:27.948955 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:27.948972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:27.951378 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:27.954840 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:27.954843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:27.954846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:27.955180 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:27.955609 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:27.955616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:27.955619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:27.955767 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:27.956376 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:27.956380 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:27.956519 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 07:36:27.956657 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 07:36:27.956923 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":4}
I20260926 07:36:27.957386 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.943331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.22.221.125:40505)
0926 07:36:27.943899 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:27.943905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:27.943930 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:36:27.943972 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:27.943978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:27.944162 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:27.944342 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:27.945103 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.946144 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.947357 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:27.947901 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:27.948045 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:27.951591 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:27.951602 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:27.951616 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:27.951648 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:27.954620 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:27.955370 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:27.955385 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:27.955388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:27.955496 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:27.955891 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:27.955899 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:27.955901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:27.956226 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:27.956446 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:27.956803 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 07:36:27.957017 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":424,"thread_start_us":291,"threads_started":1}
I20260926 07:36:27.960079 4505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:36:27.961443 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.960936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41340)
0926 07:36:27.961172 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.961271 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 07:36:27.961771 4457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:27.963342 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:27.962913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41350)
0926 07:36:27.963067 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:27.963139 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:36:27.963565 4505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab.
Entry for principal kudu/127.22.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab.
I20260926 07:36:27.968586 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:36993
--webserver_interface=127.22.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab
--principal=kudu/127.22.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43533,127.22.221.125:40505,127.22.221.124:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:28.012965 4505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 1
I20260926 07:36:28.014279 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.013740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41358)
0926 07:36:28.013963 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.014114 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 07:36:28.015376 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.014957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41374)
0926 07:36:28.015135 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.015200 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:36:28.015583 4505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:28.083829 4505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 2
I20260926 07:36:28.084998 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.084527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41386)
0926 07:36:28.084735 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.084836 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:36:28.086066 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.085718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41402)
0926 07:36:28.085872 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.085947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:36:28.086319 4505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:28.086294 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } attempt: 5
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:36:28.100132 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.086992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.22.221.125:40505)
0926 07:36:28.087151 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.087160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.087180 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:36:28.087439 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.087444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.087516 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.087625 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.088389 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.089088 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.090263 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.090689 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.090825 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.094489 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.094494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.094497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.094531 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.097381 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.098068 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.098074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.098078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.098203 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.098602 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.098608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.098611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.098904 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.099076 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.099546 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.099766 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:36:28.100131 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.086902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60800)
0926 07:36:28.087092 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.087110 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:28.087302 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:28.087656 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:28.087660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:28.088054 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:28.088225 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.088233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.089239 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.089244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.090111 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.090122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.090838 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.090844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.091053 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.091822 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:28.091868 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:28.094259 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:28.097549 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.097554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.097558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.097938 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.098327 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.098331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.098335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.098486 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.099037 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:28.099041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:28.099206 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 07:36:28.099446 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 07:36:28.099777 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:36:28.101533 4431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:36:28.102706 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.102303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41410)
0926 07:36:28.102493 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.102576 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 07:36:28.103824 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.103516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41422)
0926 07:36:28.103648 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.103724 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:36:28.104110 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:28.155385 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 1
I20260926 07:36:28.156726 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.156244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41430)
0926 07:36:28.156454 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.156546 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:36:28.157809 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.157485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41446)
0926 07:36:28.157630 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.157703 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:36:28.158071 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:28.177564 4505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 3
I20260926 07:36:28.178828 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.178240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41458)
0926 07:36:28.178502 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.178615 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 07:36:28.180075 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.179550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41472)
0926 07:36:28.179808 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.179910 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260926 07:36:28.180320 4505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:28.226333 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 2
I20260926 07:36:28.227639 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.227082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41486)
0926 07:36:28.227373 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.227474 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:36:28.228766 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.228436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41494)
0926 07:36:28.228581 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.228658 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 07:36:28.229058 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
W20260926 07:36:28.305719 4513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:28.305985 4513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:28.306071 4513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:28.313262 4513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:28.313367 4513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:28.313417 4513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:28.313495 4513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:28.313598 4513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:28.323166 4513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43533,127.22.221.125:40505,127.22.221.124:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab
--principal=kudu/127.22.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:28.323552 4505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 4
I20260926 07:36:28.324631 4513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:28.324748 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.324187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41498)
0926 07:36:28.324441 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.324556 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 07:36:28.325759 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.325390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41512)
0926 07:36:28.325555 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.325643 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 07:36:28.325991 4505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
I20260926 07:36:28.326200 4513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:36:28.335310 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 3
I20260926 07:36:28.336308 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.335921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41520)
0926 07:36:28.336082 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.336159 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:36:28.337622 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.337178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41528)
0926 07:36:28.337370 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.337481 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260926 07:36:28.337909 4431 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:36993: Network error: Client connection negotiation failed: client connection to 127.22.221.124:36993: connect: Connection refused (error 111)
W20260926 07:36:28.338794 4518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:28.338829 4519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:28.339157 4513 server_base.cc:1034] running on GCE node
W20260926 07:36:28.339128 4521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:28.340268 4513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:28.341616 4513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:28.342847 4513 hybrid_clock.cc:648] HybridClock initialized: now 1790408188342801 us; error 49 us; skew 500 ppm
I20260926 07:36:28.343197 4513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:28.346401 4513 init.cc:377] Logged in from keytab as kudu/127.22.221.124@KRBTEST.COM (short username kudu)
I20260926 07:36:28.348069 4513 webserver.cc:460] Webserver started at http://127.22.221.124:38409/ using document root <none> and password file <none>
I20260926 07:36:28.348785 4513 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:28.348941 4513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:28.349254 4513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:28.352319 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data/instance:
uuid: "9fd47dffa4b74e47afc20efb1d67b5f5"
format_stamp: "Formatted at 2026-09-26 07:36:28 on dist-test-slave-qlq8"
server_key: "ba68edc29a66e09d64c1e7ac89d56db0"
server_key_iv: "0ff4a9a6e68d51cae551635cc5fab802"
server_key_version: "encryptionkey@0"
I20260926 07:36:28.353127 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal/instance:
uuid: "9fd47dffa4b74e47afc20efb1d67b5f5"
format_stamp: "Formatted at 2026-09-26 07:36:28 on dist-test-slave-qlq8"
server_key: "ba68edc29a66e09d64c1e7ac89d56db0"
server_key_iv: "0ff4a9a6e68d51cae551635cc5fab802"
server_key_version: "encryptionkey@0"
I20260926 07:36:28.358250 4513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:36:28.361673 4528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:28.362990 4513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:36:28.363162 4513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal
uuid: "9fd47dffa4b74e47afc20efb1d67b5f5"
format_stamp: "Formatted at 2026-09-26 07:36:28 on dist-test-slave-qlq8"
server_key: "ba68edc29a66e09d64c1e7ac89d56db0"
server_key_iv: "0ff4a9a6e68d51cae551635cc5fab802"
server_key_version: "encryptionkey@0"
I20260926 07:36:28.363417 4513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:28.403424 4513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:28.407858 4513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:28.408160 4513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:28.426584 4513 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.124:36993
I20260926 07:36:28.426621 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.124:36993 every 8 connection(s)
I20260926 07:36:28.428056 4513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
I20260926 07:36:28.431600 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4513
I20260926 07:36:28.431839 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407989013554-23413-0/minicluster-data/master-2/wal/instance
I20260926 07:36:28.432243 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:28.432425 23413 external_mini_cluster.cc:1428] Setting key 9042c7e8b04ccab74eebcd86a3ff479a
I20260926 07:36:28.442047 4580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:28.456995 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.434306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.22.221.124:36993)
0926 07:36:28.435128 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.435139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.435182 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:36:28.437031 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.437052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.437427 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.437735 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.439627 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.440956 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.442942 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.443751 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.443880 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.448858 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.448867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.448878 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.448916 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.452220 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.452911 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.452920 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.452935 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.453086 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.453520 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.453528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.453532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.454159 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.454481 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.454996 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.455501 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":386,"threads_started":1}
I20260926 07:36:28.460872 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.445137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:43533 (local address 127.0.0.1:38740)
0926 07:36:28.446087 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.446127 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:28.446276 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:28.446699 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:28.446704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:28.447365 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:28.447626 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.447634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.448710 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.448722 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.450070 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.450082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.451029 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.451036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.451211 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.451921 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:28.451938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:28.454029 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:28.457746 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.457755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.457761 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.458339 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.458895 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.458899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.458901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.459023 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.459665 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:28.459670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:28.459918 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 07:36:28.460125 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:36:28.460451 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":564,"threads_started":1}
I20260926 07:36:28.460893 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.445179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.22.221.126:43533)
0926 07:36:28.445737 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.445743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.446235 (+ 492us) server_negotiation.cc:407] Connection header received
0926 07:36:28.446407 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.446414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.446514 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.446647 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.447783 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.448543 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.450233 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.450825 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.450995 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.454219 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.454226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.454234 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.454276 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.457622 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.458585 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.458591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.458594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.458765 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.459141 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.459148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.459153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.459535 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.459703 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.460233 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.460452 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":255,"threads_started":1}
I20260926 07:36:28.465952 4580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:36:28.477347 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.460676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.22.221.126:43533)
0926 07:36:28.461523 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.461527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.461542 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:36:28.461585 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.461588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.461657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.461775 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.463192 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.463967 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.465258 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.465932 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.466060 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.471683 (+ 5623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.471689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.471694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.471747 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.474499 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.475210 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.475217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.475220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.475413 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.475794 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.475798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.475802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.476162 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.476472 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.476789 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.477016 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":4729,"threads_started":1}
I20260926 07:36:28.481422 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.466713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:40505 (local address 127.0.0.1:60816)
0926 07:36:28.466948 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.466968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:28.467065 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:28.467888 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:28.467893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:28.468386 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:28.468598 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.468610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.470132 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.470141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.471577 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.471591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.472583 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.472590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.472784 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.473340 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:28.473368 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:28.475333 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:28.478801 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.478805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.478810 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.479370 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.479779 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.479785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.479788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.479974 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.480462 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:28.480469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:28.480659 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 07:36:28.480814 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:36:28.481080 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:36:28.481427 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.467345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.22.221.125:40505)
0926 07:36:28.467534 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.467562 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.467582 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:36:28.467657 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.467666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.467744 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.467858 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.468787 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.469997 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.471726 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.472430 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.472524 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.475521 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.475527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.475532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.475568 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.478667 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.479500 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.479506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.479509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.479664 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.480090 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.480094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.480096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.480333 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.480448 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.480921 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.481071 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:36:28.482400 4580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408188, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:36:28.494967 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.479020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.22.221.125:40505)
0926 07:36:28.479623 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.479627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.479646 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:36:28.479683 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.479686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.479742 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.479819 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.481323 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.482430 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.484041 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.484681 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.484769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.489348 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.489358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.489366 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.489403 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.492061 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.492762 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.492768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.492771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.492918 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.493291 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.493296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.493299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.493666 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.493878 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.494309 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.494548 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
I20260926 07:36:28.496351 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.483375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.22.221.124:36993)
0926 07:36:28.483545 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.483551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.483568 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:36:28.483612 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.483617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.483680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.483780 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.484584 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.485551 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.486681 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.487431 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.487559 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.490569 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.490574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.490577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.490603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.493613 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.494402 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.494408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.494414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.494549 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.494933 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.494940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.494946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.495343 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.495496 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.495835 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.496078 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:36:28.496701 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.483103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41540)
0926 07:36:28.483291 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.483309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:28.483407 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:28.483833 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:28.483836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:28.484259 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:28.484438 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.484450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.485679 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.485683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.486536 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.486547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.487618 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.487623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.487761 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.488346 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:28.488362 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:28.490401 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:28.493774 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.493779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.493782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.494269 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.494675 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.494678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.494682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.494813 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.495462 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:28.495466 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:28.495608 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 07:36:28.495754 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 07:36:28.496291 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":194}
I20260926 07:36:28.500099 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:36:28.501717 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: Bootstrap starting.
I20260926 07:36:28.505004 4580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:28.506446 4580 log.cc:824] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:28.512676 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: No bootstrap required, opened a new log
I20260926 07:36:28.515158 4431 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 4
I20260926 07:36:28.524642 4580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.524958 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.505021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.22.221.126:43533)
0926 07:36:28.505312 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.505320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.506119 (+ 799us) server_negotiation.cc:407] Connection header received
0926 07:36:28.506396 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.506404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.506494 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.506571 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.509699 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.510943 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.513179 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.513816 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.513916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.518037 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.518044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.518048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.518087 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.521657 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.522524 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.522529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.522533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.522688 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.523162 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.523167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.523171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.523546 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.523782 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.524374 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.524595 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260926 07:36:28.525079 4580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:28.525220 4580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd47dffa4b74e47afc20efb1d67b5f5, State: Initialized, Role: FOLLOWER
I20260926 07:36:28.526005 4580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.533125 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.505253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.22.221.124:36993)
0926 07:36:28.505509 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.505515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.512467 (+ 6952us) server_negotiation.cc:407] Connection header received
0926 07:36:28.512996 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.513004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.513103 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.513194 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.516010 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.517400 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.519963 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.521037 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.521165 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.525078 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.525087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.525094 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.525142 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.529359 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.530409 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.530418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.530424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.530591 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.531193 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.531200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.531207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.531585 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.531895 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.532495 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.532705 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:36:28.533821 4597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.534242 4597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.536355 4580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:36:28.544732 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.506533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.22.221.125:40505)
0926 07:36:28.507067 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.507075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.526126 (+ 19051us) server_negotiation.cc:407] Connection header received
0926 07:36:28.526546 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.526555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.526665 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.526754 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.528699 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.530093 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.532425 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.533305 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.533411 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.537384 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.537391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.537396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.537437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.541225 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.542454 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.542460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.542463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.542622 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.543116 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.543122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.543127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.543497 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.543766 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.544108 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.544351 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
I20260926 07:36:28.549865 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.515705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41548)
0926 07:36:28.516031 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.516050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:28.516145 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:28.533854 (+ 17709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:28.533859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:28.534330 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:28.534562 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.534576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.536038 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.536044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.537057 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.537069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.538186 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.538192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.538383 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.539473 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:28.539509 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:28.542153 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:28.546971 (+ 4818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.546976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.546978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.547456 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.547950 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.547958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.547964 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.548179 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.548979 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:28.548985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:28.549139 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 07:36:28.549296 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 07:36:28.549611 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260926 07:36:28.550596 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.516470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.22.221.124:36993)
0926 07:36:28.533491 (+ 17021us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.533497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.533516 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:36:28.533563 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.533566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.533631 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.533832 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.534730 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.535854 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.537214 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.538001 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.538169 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.542390 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.542399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.542404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.542448 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.546792 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.547642 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.547647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.547653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.547802 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.548318 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.548323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.548326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.548750 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.548898 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.550175 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.550336 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16873,"spinlock_wait_cycles":19328,"thread_start_us":16842,"threads_started":1}
I20260926 07:36:28.556350 4431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: Bootstrap starting.
W20260926 07:36:28.559319 4608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:36:28.559445 4608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:36:28.560835 4431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:28.562121 4431 log.cc:824] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:28.553913 4513 master_runner.cc:186] Error getting master registration for 127.22.221.126:43533: OK, instance_id { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" instance_seqno: 1790408187373972 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:36:28.564556 4513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:36:28.565410 4431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: No bootstrap required, opened a new log
I20260926 07:36:28.576292 4431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.576691 4431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:28.576812 4431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5ec601be374c5884ce4a3e013dd089, State: Initialized, Role: FOLLOWER
I20260926 07:36:28.577468 4431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.578385 4611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.578729 4611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.579890 4431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:36:28.590277 4622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:36:28.590420 4622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:36:28.589660 4364 master_runner.cc:186] Error getting master registration for 127.22.221.125:40505: OK, instance_id { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" instance_seqno: 1790408187894793 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:36:28.600849 4364 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:36:28.613200 4385 tablet.cc:2369] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:36:28.625232 4505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } attempt: 5
Sep 26 07:36:28 dist-test-slave-qlq8 krb5kdc[4351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408187, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:36:28.643368 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.625930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:36993 (local address 127.0.0.1:41564)
0926 07:36:28.626221 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:28.626242 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:28.626377 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:28.626805 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:28.626810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:28.627375 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:28.627590 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.627603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.629575 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.629581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.631058 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:28.631076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.632263 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.632269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:28.632497 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.633207 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:28.633232 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:28.636117 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:28.640422 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.640428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.640432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.640943 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.641387 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:28.641392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:28.641396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:28.641571 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:28.642265 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:28.642270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:28.642433 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 07:36:28.642594 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 07:36:28.642940 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 07:36:28.644174 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:28.626044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.22.221.124:36993)
0926 07:36:28.626226 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:28.626233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:28.626333 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:36:28.626522 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:28.626527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:28.626616 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:28.626737 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:28.627764 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.628947 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.631286 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:28.632049 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:28.632190 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:28.636332 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:28.636339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:28.636344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:28.636388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:28.640245 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.641090 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.641097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.641101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.641257 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:28.641691 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:28.641697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:28.641700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:28.642063 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:28.642203 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:28.643448 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0926 07:36:28.643701 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:36:28.650146 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: Bootstrap starting.
I20260926 07:36:28.654536 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:28.655864 4505 log.cc:824] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:28.659189 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: No bootstrap required, opened a new log
W20260926 07:36:28.660238 4534 tablet.cc:2369] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:36:28.671119 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.671569 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:28.671694 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab04c3ce8b1f4a9bac99b4768ee2d292, State: Initialized, Role: FOLLOWER
I20260926 07:36:28.672406 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.673388 4625 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.673835 4625 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.674719 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:28.676677 4611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:36:28.676980 4611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.678597 4611 leader_election.cc:288] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab04c3ce8b1f4a9bac99b4768ee2d292 (127.22.221.125:40505), 9fd47dffa4b74e47afc20efb1d67b5f5 (127.22.221.124:36993)
I20260926 07:36:28.680154 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e5ec601be374c5884ce4a3e013dd089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" is_pre_election: true
I20260926 07:36:28.680703 4480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5ec601be374c5884ce4a3e013dd089 in term 0.
I20260926 07:36:28.681413 4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e5ec601be374c5884ce4a3e013dd089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" is_pre_election: true
I20260926 07:36:28.681988 4555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5ec601be374c5884ce4a3e013dd089 in term 0.
I20260926 07:36:28.682813 4380 leader_election.cc:302] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5ec601be374c5884ce4a3e013dd089, ab04c3ce8b1f4a9bac99b4768ee2d292; no voters:
I20260926 07:36:28.683779 4611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:36:28.683964 4611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:36:28.684068 4611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:28.690753 4611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.693825 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e5ec601be374c5884ce4a3e013dd089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292"
I20260926 07:36:28.694056 4480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:28.697530 4480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5ec601be374c5884ce4a3e013dd089 in term 1.
I20260926 07:36:28.698071 4380 leader_election.cc:302] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5ec601be374c5884ce4a3e013dd089, ab04c3ce8b1f4a9bac99b4768ee2d292; no voters:
I20260926 07:36:28.701056 4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e5ec601be374c5884ce4a3e013dd089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5"
I20260926 07:36:28.701278 4555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:28.701797 4632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:28.705163 4555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5ec601be374c5884ce4a3e013dd089 in term 1.
I20260926 07:36:28.705807 4611 leader_election.cc:288] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [CANDIDATE]: Term 1 election: Requested vote from peers ab04c3ce8b1f4a9bac99b4768ee2d292 (127.22.221.125:40505), 9fd47dffa4b74e47afc20efb1d67b5f5 (127.22.221.124:36993)
I20260926 07:36:28.706739 4632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [term 1 LEADER]: Becoming Leader. State: Replica: 4e5ec601be374c5884ce4a3e013dd089, State: Running, Role: LEADER
I20260926 07:36:28.707458 4632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } }
I20260926 07:36:28.724097 4611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5ec601be374c5884ce4a3e013dd089. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.724493 4611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:28.735704 4640 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:36:28.735895 4639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:28.738270 4639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:28.753880 4480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [term 1 FOLLOWER]: Refusing update from remote peer 4e5ec601be374c5884ce4a3e013dd089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:28.753880 4555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [term 1 FOLLOWER]: Refusing update from remote peer 4e5ec601be374c5884ce4a3e013dd089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:28.754889 4633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:28.755722 4633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:28.781581 4625 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5ec601be374c5884ce4a3e013dd089. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.782008 4625 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.787990 4597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5ec601be374c5884ce4a3e013dd089. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.788306 4597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.791473 4597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.791743 4597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.792796 4625 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.793075 4625 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:36:28.793236 4643 mvcc.cc:205] Tried to move back new op lower bound from 7333511941123993600 to 7333511940951994368. Current Snapshot: MvccSnapshot[applied={T|T < 7333511941123993600}]
I20260926 07:36:28.798223 4611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.798520 4611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:28.798795 4611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e5ec601be374c5884ce4a3e013dd089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5ec601be374c5884ce4a3e013dd089" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43533 } } peers { permanent_uuid: "ab04c3ce8b1f4a9bac99b4768ee2d292" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 40505 } } peers { permanent_uuid: "9fd47dffa4b74e47afc20efb1d67b5f5" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 36993 } } }
I20260926 07:36:28.799052 4611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:28.800428 4639 catalog_manager.cc:1347] Generated new cluster ID: 2f9e21019cb94550b251c44642a0496e
I20260926 07:36:28.800532 4639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:28.801465 4642 mvcc.cc:205] Tried to move back new op lower bound from 7333511941123993600 to 7333511940951994368. Current Snapshot: MvccSnapshot[applied={T|T < 7333511941123993600}]
I20260926 07:36:28.821019 4639 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:36:28.822502 4639 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:28.844300 4639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e5ec601be374c5884ce4a3e013dd089: Generated new TSK 0
I20260926 07:36:28.845270 4639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:36:29.561339 4608 catalog_manager.cc:1259] Loaded cluster ID: 2f9e21019cb94550b251c44642a0496e
I20260926 07:36:29.561447 4608 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: loading cluster ID for follower catalog manager: success
I20260926 07:36:29.563925 4608 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: acquiring CA information for follower catalog manager: success
I20260926 07:36:29.565646 4608 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fd47dffa4b74e47afc20efb1d67b5f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:36:29.740278 4640 catalog_manager.cc:1259] Loaded cluster ID: 2f9e21019cb94550b251c44642a0496e
I20260926 07:36:29.740404 4640 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: loading cluster ID for follower catalog manager: success
I20260926 07:36:29.742975 4640 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: acquiring CA information for follower catalog manager: success
I20260926 07:36:29.744853 4640 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab04c3ce8b1f4a9bac99b4768ee2d292: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:36:29.931710 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4364
I20260926 07:36:29.933634 4426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:30.045578 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4364
W20260926 07:36:30.074179 4530 connection.cc:535] server connection from 127.0.0.1:41548 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:36:30.074828 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4438
I20260926 07:36:30.076215 4500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:30.190588 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4438
I20260926 07:36:30.210258 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4513
I20260926 07:36:30.211421 4575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:30.318738 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4513
2026-09-26T07:36:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4783 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:36:30 dist-test-slave-qlq8 krb5kdc[4658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:36:30 dist-test-slave-qlq8 krb5kdc[4658](info): set up 2 sockets
Sep 26 07:36:30 dist-test-slave-qlq8 krb5kdc[4658](info): commencing operation
krb5kdc: starting...
W20260926 07:36:31.443914 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:36:31 dist-test-slave-qlq8 krb5kdc[4658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:36:31.668917 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45095
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:45095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:31.994889 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:31.995182 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:31.995399 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:32.002907 4671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:32.003010 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:32.003116 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:32.003213 4671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:32.003322 4671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:32.013383 4671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:45095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:32.014706 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:32.016109 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:32.027992 4677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:32.028079 4676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:32.028064 4679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:32.029068 4671 server_base.cc:1034] running on GCE node
I20260926 07:36:32.029807 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:32.031200 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:32.032491 4671 hybrid_clock.cc:648] HybridClock initialized: now 1790408192032446 us; error 46 us; skew 500 ppm
I20260926 07:36:32.032841 4671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:32 dist-test-slave-qlq8 krb5kdc[4658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408192, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:32.036064 4671 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:32.037533 4671 webserver.cc:460] Webserver started at http://127.22.221.126:36987/ using document root <none> and password file <none>
I20260926 07:36:32.038214 4671 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:32.038360 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:32.038668 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:32.041663 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "66b72634dca946e0a6d9190e37b8c14f"
format_stamp: "Formatted at 2026-09-26 07:36:32 on dist-test-slave-qlq8"
server_key: "a2d0da20a5dc14467979ec092e46048c"
server_key_iv: "f06e795f1829ef148f22eba199a9f611"
server_key_version: "encryptionkey@0"
I20260926 07:36:32.042421 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "66b72634dca946e0a6d9190e37b8c14f"
format_stamp: "Formatted at 2026-09-26 07:36:32 on dist-test-slave-qlq8"
server_key: "a2d0da20a5dc14467979ec092e46048c"
server_key_iv: "f06e795f1829ef148f22eba199a9f611"
server_key_version: "encryptionkey@0"
I20260926 07:36:32.047508 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 07:36:32.051012 4686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:32.052369 4671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:32.052558 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "66b72634dca946e0a6d9190e37b8c14f"
format_stamp: "Formatted at 2026-09-26 07:36:32 on dist-test-slave-qlq8"
server_key: "a2d0da20a5dc14467979ec092e46048c"
server_key_iv: "f06e795f1829ef148f22eba199a9f611"
server_key_version: "encryptionkey@0"
I20260926 07:36:32.052764 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:32.068977 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:32.073329 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:32.073626 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:32.092563 4671 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:45095
I20260926 07:36:32.092576 4737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:45095 every 8 connection(s)
I20260926 07:36:32.094028 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:32.097774 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:32.100349 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4671
I20260926 07:36:32.100591 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:36:32.101279 23413 external_mini_cluster.cc:1428] Setting key 88faf00a8ff63e6c5353c623046c2ea6
I20260926 07:36:32.110713 4738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f: Bootstrap starting.
I20260926 07:36:32.114269 4738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:32.115311 4738 log.cc:824] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:36:32 dist-test-slave-qlq8 krb5kdc[4658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:32.118767 4738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f: No bootstrap required, opened a new log
I20260926 07:36:32.126576 4741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:32.103341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.22.221.126:45095)
0926 07:36:32.104111 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:32.104126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:32.104173 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:36:32.106517 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:32.106547 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:32.107092 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:32.107563 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:32.109015 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:32.110976 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:32.112432 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:32.113451 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:32.113643 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:32.118672 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:32.118681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:32.118732 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:32.118774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:32.122107 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:32.122982 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:32.122989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:32.122995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:32.123172 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:32.123733 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:32.123738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:32.123740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:32.124318 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:32.124570 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:32.125067 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 07:36:32.125493 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":332,"threads_started":1}
I20260926 07:36:32.127660 4738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b72634dca946e0a6d9190e37b8c14f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45095 } }
I20260926 07:36:32.128008 4738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:32.128119 4738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b72634dca946e0a6d9190e37b8c14f, State: Initialized, Role: FOLLOWER
I20260926 07:36:32.128757 4738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b72634dca946e0a6d9190e37b8c14f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45095 } }
I20260926 07:36:32.129024 4738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:32.129168 4738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:32.129346 4738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:32.130781 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:36:32.131940 4738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b72634dca946e0a6d9190e37b8c14f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45095 } }
I20260926 07:36:32.132416 4738 leader_election.cc:302] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b72634dca946e0a6d9190e37b8c14f; no voters:
I20260926 07:36:32.133035 4738 leader_election.cc:288] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:36:32.133188 4743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:32.134599 4743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [term 1 LEADER]: Becoming Leader. State: Replica: 66b72634dca946e0a6d9190e37b8c14f, State: Running, Role: LEADER
I20260926 07:36:32.135080 4743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b72634dca946e0a6d9190e37b8c14f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45095 } }
I20260926 07:36:32.135426 4738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:32.137782 4745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b72634dca946e0a6d9190e37b8c14f. Latest consensus state: current_term: 1 leader_uuid: "66b72634dca946e0a6d9190e37b8c14f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b72634dca946e0a6d9190e37b8c14f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45095 } } }
I20260926 07:36:32.138175 4745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:32.138739 4744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b72634dca946e0a6d9190e37b8c14f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b72634dca946e0a6d9190e37b8c14f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45095 } } }
I20260926 07:36:32.139091 4744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:32.139207 4751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:32.145478 4751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:32.158146 4751 catalog_manager.cc:1347] Generated new cluster ID: 15cb93dddbd549adb9ad126dbd07a77c
I20260926 07:36:32.158264 4751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:32.179826 4751 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:36:32.181172 4751 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:32.191464 4751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b72634dca946e0a6d9190e37b8c14f: Generated new TSK 0
I20260926 07:36:32.192562 4751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:36:32.243535 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4671
I20260926 07:36:32.244695 4733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:32.342406 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4671
2026-09-26T07:36:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2018 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:36:32 dist-test-slave-qlq8 krb5kdc[4767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:36:32 dist-test-slave-qlq8 krb5kdc[4767](info): set up 2 sockets
Sep 26 07:36:32 dist-test-slave-qlq8 krb5kdc[4767](info): commencing operation
krb5kdc: starting...
W20260926 07:36:33.463861 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:36:33 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:36:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:36:33.694916 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40587
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:34.018754 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:34.019176 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:34.019390 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:34.027796 4780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:34.027902 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:34.028033 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:34.028129 4780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:34.028211 4780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:34.038054 4780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:34.039439 4780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:34.040781 4780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:34.052654 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:34.052753 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:34.052721 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:34.054116 4780 server_base.cc:1034] running on GCE node
I20260926 07:36:34.054976 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:34.056555 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:34.057766 4780 hybrid_clock.cc:648] HybridClock initialized: now 1790408194057723 us; error 43 us; skew 500 ppm
I20260926 07:36:34.058125 4780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:34 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408194, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:34.061424 4780 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:34.062865 4780 webserver.cc:460] Webserver started at http://127.22.221.126:34483/ using document root <none> and password file <none>
I20260926 07:36:34.063622 4780 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:34.063738 4780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:34.064097 4780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:34.067098 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "25ae8838b3ab40e2b2fca33613706e54"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "ec5e72cfc57983eba0576870fb3abd34"
server_key_iv: "67e206faa3a6a14bd14c9bc7cc042e45"
server_key_version: "encryptionkey@0"
I20260926 07:36:34.067865 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "25ae8838b3ab40e2b2fca33613706e54"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "ec5e72cfc57983eba0576870fb3abd34"
server_key_iv: "67e206faa3a6a14bd14c9bc7cc042e45"
server_key_version: "encryptionkey@0"
I20260926 07:36:34.072887 4780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:36:34.076388 4795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:34.077956 4780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:36:34.078104 4780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "25ae8838b3ab40e2b2fca33613706e54"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "ec5e72cfc57983eba0576870fb3abd34"
server_key_iv: "67e206faa3a6a14bd14c9bc7cc042e45"
server_key_version: "encryptionkey@0"
I20260926 07:36:34.078282 4780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:34.105202 4780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:34.109768 4780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:34.110025 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:34.129271 4780 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:40587
I20260926 07:36:34.129308 4846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:40587 every 8 connection(s)
I20260926 07:36:34.130661 4780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:34.134917 4847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:34.138162 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4780
I20260926 07:36:34.138446 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:36:34.139390 23413 external_mini_cluster.cc:1428] Setting key c67458e5ef53a9c18a7d425ad110971e
I20260926 07:36:34.148279 4847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap starting.
I20260926 07:36:34.151583 4847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:34.152665 4847 log.cc:824] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:36:34 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:34.155568 4847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: No bootstrap required, opened a new log
I20260926 07:36:34.163174 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:34.141772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.22.221.126:40587)
0926 07:36:34.142579 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:34.142592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:34.142631 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:36:34.144353 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:34.144384 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:34.144771 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:34.145174 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:34.146678 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:34.148080 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:34.149482 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:34.150176 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:34.150369 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:34.154823 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:34.154832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:34.154849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:34.154887 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:34.158476 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:34.159327 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:34.159334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:34.159340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:34.159531 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:34.160073 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:34.160079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:34.160083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:34.160705 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:34.161073 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:34.161642 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 07:36:34.162105 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":385,"threads_started":1}
I20260926 07:36:34.163563 4847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:34.164016 4847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:34.164136 4847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae8838b3ab40e2b2fca33613706e54, State: Initialized, Role: FOLLOWER
I20260926 07:36:34.164769 4847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:34.165055 4847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:34.165207 4847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:34.165386 4847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:34.167928 4847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:34.168427 4847 leader_election.cc:302] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ae8838b3ab40e2b2fca33613706e54; no voters:
I20260926 07:36:34.169045 4847 leader_election.cc:288] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:36:34.169202 4852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:34.170433 4852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 LEADER]: Becoming Leader. State: Replica: 25ae8838b3ab40e2b2fca33613706e54, State: Running, Role: LEADER
I20260926 07:36:34.171288 4847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:34.171149 4852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:34.174243 4854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ae8838b3ab40e2b2fca33613706e54. Latest consensus state: current_term: 1 leader_uuid: "25ae8838b3ab40e2b2fca33613706e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } } }
I20260926 07:36:34.174712 4854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:34.174664 4853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ae8838b3ab40e2b2fca33613706e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } } }
I20260926 07:36:34.175104 4853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:34.175910 4861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:34.182348 4861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:34.196451 4861 catalog_manager.cc:1347] Generated new cluster ID: aee55d59ebeb418f8ec94476b8001a48
I20260926 07:36:34.196671 4861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:34.212393 4861 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:36:34.213672 4861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:34.219267 4861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Generated new TSK 0
I20260926 07:36:34.220135 4861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:36:34.279086 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:40587
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:34.616199 4873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:34.616468 4873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:34.616649 4873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:34.623827 4873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:34.623937 4873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:34.624130 4873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:36:34.633677 4873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:34.634963 4873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:34.636334 4873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:34.650200 4879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:34.650233 4878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:34.650303 4881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:34.652520 4873 server_base.cc:1034] running on GCE node
I20260926 07:36:34.653142 4873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:34.654176 4873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:34.655422 4873 hybrid_clock.cc:648] HybridClock initialized: now 1790408194655366 us; error 45 us; skew 500 ppm
I20260926 07:36:34.655746 4873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:34 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408194, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:34.658819 4873 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:36:34.660374 4873 webserver.cc:460] Webserver started at http://127.22.221.65:43283/ using document root <none> and password file <none>
I20260926 07:36:34.661129 4873 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:34.661296 4873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:34.661600 4873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:34.664584 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "5cde7fca027b4c41a8be28ed02bf5b65"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "8b9bbdfad55107964a7212a375d4c24b"
server_key_iv: "d972e4f3e8d63705ff1ab1b1808234cb"
server_key_version: "encryptionkey@0"
I20260926 07:36:34.665373 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "5cde7fca027b4c41a8be28ed02bf5b65"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "8b9bbdfad55107964a7212a375d4c24b"
server_key_iv: "d972e4f3e8d63705ff1ab1b1808234cb"
server_key_version: "encryptionkey@0"
I20260926 07:36:34.670502 4873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 07:36:34.673904 4888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:34.675436 4873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:34.675601 4873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "5cde7fca027b4c41a8be28ed02bf5b65"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "8b9bbdfad55107964a7212a375d4c24b"
server_key_iv: "d972e4f3e8d63705ff1ab1b1808234cb"
server_key_version: "encryptionkey@0"
I20260926 07:36:34.675778 4873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:34.703702 4873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:34.707512 4873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:34.707819 4873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:34.708712 4873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:34.710009 4873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:34.710083 4873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:34.710163 4873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:34.710208 4873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:34.743903 4873 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:39963
I20260926 07:36:34.743956 5000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:39963 every 8 connection(s)
I20260926 07:36:34.745200 4873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:36:34.753013 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 4873
I20260926 07:36:34.753240 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:36:34.753892 23413 external_mini_cluster.cc:1428] Setting key a1b197d0ff7b2dbc605838895ffee861
Sep 26 07:36:34 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408194, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:34.767146 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:34.748006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:59425 (local address 127.22.221.126:40587)
0926 07:36:34.748438 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:34.748443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:34.749372 (+ 929us) server_negotiation.cc:407] Connection header received
0926 07:36:34.751207 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:34.751212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:34.751346 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:34.751468 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:34.752997 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:34.753894 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:34.755795 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:34.756344 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:34.756503 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:34.760067 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:34.760076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:34.760085 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:34.760130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:34.763680 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:34.764718 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:34.764722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:34.764724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:34.764842 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:34.765360 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:34.765364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:34.765366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:34.765648 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:34.765790 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:34.766701 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 07:36:34.766886 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":212,"threads_started":1}
I20260926 07:36:34.768791 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:34.748538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:59425)
0926 07:36:34.749209 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:34.749253 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:34.750935 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:34.751723 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:34.751731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:34.752241 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:34.752825 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:34.752843 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:34.754083 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:34.754087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:34.755618 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:34.755640 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:34.756603 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:34.756608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:34.756811 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:34.757558 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:34.757580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:34.759836 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:34.763860 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:34.763872 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:34.763893 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:34.764587 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:34.764958 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:34.764965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:34.764972 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:34.765220 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:34.765791 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:34.765802 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:34.766265 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 07:36:34.766596 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 07:36:34.767190 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260926 07:36:34.770457 5001 heartbeater.cc:347] Connected to a master server at 127.22.221.126:40587
I20260926 07:36:34.770864 5001 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:34.771783 5001 heartbeater.cc:511] Master 127.22.221.126:40587 requested a full tablet report, sending...
I20260926 07:36:34.774055 4812 ts_manager.cc:194] Registered new tserver with Master: 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65:39963)
I20260926 07:36:34.775893 4812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:59425
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:36:34.808823 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:40587
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:35.139990 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:35.140416 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:35.140558 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:35.151889 5007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:35.151997 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:35.152165 5007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:36:35.161924 5007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:35.163166 5007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:35.164544 5007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:35.177938 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:35.177860 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:35.178304 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:35.178809 5007 server_base.cc:1034] running on GCE node
I20260926 07:36:35.179492 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:35.180514 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:35.181792 5007 hybrid_clock.cc:648] HybridClock initialized: now 1790408195181710 us; error 54 us; skew 500 ppm
I20260926 07:36:35.182189 5007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:35 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408195, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:35.185410 5007 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:36:35.186865 5007 webserver.cc:460] Webserver started at http://127.22.221.66:36463/ using document root <none> and password file <none>
I20260926 07:36:35.187579 5007 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:35.187680 5007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:35.187948 5007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:35.190718 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "45ef4776466046418c871e49e775ca5a"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "14df5bea1539a0d5048bf6bf0f4c6232"
server_key_iv: "41add901c5e1d87f76689b0ad271c8ba"
server_key_version: "encryptionkey@0"
I20260926 07:36:35.191514 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "45ef4776466046418c871e49e775ca5a"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "14df5bea1539a0d5048bf6bf0f4c6232"
server_key_iv: "41add901c5e1d87f76689b0ad271c8ba"
server_key_version: "encryptionkey@0"
I20260926 07:36:35.196431 5007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:36:35.199935 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:35.201370 5007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:36:35.201494 5007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "45ef4776466046418c871e49e775ca5a"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "14df5bea1539a0d5048bf6bf0f4c6232"
server_key_iv: "41add901c5e1d87f76689b0ad271c8ba"
server_key_version: "encryptionkey@0"
I20260926 07:36:35.201644 5007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:35.237946 5007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:35.241873 5007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:35.242110 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:35.242990 5007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:35.244256 5007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:35.244331 5007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:35.244418 5007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:35.244462 5007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:35.278364 5007 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:46843
I20260926 07:36:35.278403 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:46843 every 8 connection(s)
I20260926 07:36:35.279768 5007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:36:35.282018 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5007
I20260926 07:36:35.282215 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:36:35.282840 23413 external_mini_cluster.cc:1428] Setting key 3ef571c03f138aff2ea1dc9525664818
Sep 26 07:36:35 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408195, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:35.304431 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.282514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:40951 (local address 127.22.221.126:40587)
0926 07:36:35.283008 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:35.283015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:35.284257 (+ 1242us) server_negotiation.cc:407] Connection header received
0926 07:36:35.286868 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:35.286873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:35.286946 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:35.287044 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:35.289201 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.289978 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.292574 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.293045 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.293154 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.296904 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:35.296914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:35.296922 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:35.296969 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:35.300883 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:35.301640 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:35.301648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:35.301654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:35.301839 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:35.302313 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:35.302320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:35.302326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:35.302708 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:35.302876 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:35.303626 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 07:36:35.303910 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
I20260926 07:36:35.305176 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.283212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:40951)
0926 07:36:35.284110 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:35.284153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:35.286516 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:35.287370 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:35.287384 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:35.288176 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:35.289000 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.289030 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.290148 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.290157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.292402 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.292422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.293287 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.293296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.293626 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.294411 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:35.294438 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:35.296630 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:35.301032 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:35.301038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:35.301051 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:35.301495 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:35.301985 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:35.301989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:35.301991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:35.302177 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:35.302833 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:35.302839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:35.303159 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 07:36:35.303475 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 07:36:35.303898 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":360,"threads_started":1}
I20260926 07:36:35.306752 5135 heartbeater.cc:347] Connected to a master server at 127.22.221.126:40587
I20260926 07:36:35.307121 5135 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:35.308081 5135 heartbeater.cc:511] Master 127.22.221.126:40587 requested a full tablet report, sending...
I20260926 07:36:35.309846 4812 ts_manager.cc:194] Registered new tserver with Master: 45ef4776466046418c871e49e775ca5a (127.22.221.66:46843)
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:36:35.310839 4812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:40951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:36:35.340654 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:40587
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:35.663566 5141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:35.663833 5141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:35.664040 5141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:35.671119 5141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:35.671276 5141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:35.671479 5141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:36:35.681185 5141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:35.682497 5141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:35.683969 5141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:35.697706 5146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:35.697758 5147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:35.697667 5149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:35.698911 5141 server_base.cc:1034] running on GCE node
I20260926 07:36:35.699605 5141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:35.700598 5141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:35.701818 5141 hybrid_clock.cc:648] HybridClock initialized: now 1790408195701763 us; error 52 us; skew 500 ppm
I20260926 07:36:35.702142 5141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:35 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408195, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:35.705345 5141 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:36:35.706790 5141 webserver.cc:460] Webserver started at http://127.22.221.67:33601/ using document root <none> and password file <none>
I20260926 07:36:35.707508 5141 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:35.707612 5141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:35.707862 5141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:35.710748 5141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "9ab39e6c0db14929a891d1ac86a5d917"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "8410112c2bb075212d02bcb4baf48dcb"
server_key_iv: "f6e1f068b5d8672e2f334735e42e3f91"
server_key_version: "encryptionkey@0"
I20260926 07:36:35.711597 5141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "9ab39e6c0db14929a891d1ac86a5d917"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "8410112c2bb075212d02bcb4baf48dcb"
server_key_iv: "f6e1f068b5d8672e2f334735e42e3f91"
server_key_version: "encryptionkey@0"
I20260926 07:36:35.716720 5141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:36:35.720296 5156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:35.721625 5141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:36:35.721801 5141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "9ab39e6c0db14929a891d1ac86a5d917"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "8410112c2bb075212d02bcb4baf48dcb"
server_key_iv: "f6e1f068b5d8672e2f334735e42e3f91"
server_key_version: "encryptionkey@0"
I20260926 07:36:35.722013 5141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:35.762054 5141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:35.766114 5141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:35.766456 5141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:35.767534 5141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:35.768961 5141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:35.769068 5141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:35.769220 5141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:35.769295 5141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:35.779479 5001 heartbeater.cc:503] Master 127.22.221.126:40587 was elected leader, sending a full tablet report...
I20260926 07:36:35.804366 5141 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:37047
I20260926 07:36:35.804410 5268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:37047 every 8 connection(s)
I20260926 07:36:35.805748 5141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:36:35.813295 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5141
I20260926 07:36:35.813519 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:36:35.814091 23413 external_mini_cluster.cc:1428] Setting key ae3a3b06019a5f0b0728969e90dea7e1
Sep 26 07:36:35 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408195, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:35.826602 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.808472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:53569 (local address 127.22.221.126:40587)
0926 07:36:35.808917 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:35.808925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:35.809865 (+ 940us) server_negotiation.cc:407] Connection header received
0926 07:36:35.811791 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:35.811796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:35.811901 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:35.812002 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:35.813427 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.814583 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.816723 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.817302 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.817400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.820858 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:35.820863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:35.820867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:35.820910 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:35.823738 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:35.824491 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:35.824494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:35.824496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:35.824611 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:35.824998 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:35.825002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:35.825004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:35.825306 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:35.825430 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:35.826111 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 07:36:35.826319 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":201,"threads_started":1}
I20260926 07:36:35.827692 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.809028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:53569)
0926 07:36:35.809696 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:35.809736 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:35.811513 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:35.812196 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:35.812204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:35.812689 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:35.813241 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.813259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.814761 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.814767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.816589 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.816603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.817525 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.817530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.817723 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.818345 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:35.818381 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:35.820643 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:35.823878 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:35.823885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:35.823901 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:35.824378 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:35.824699 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:35.824702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:35.824704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:35.824880 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:35.825420 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:35.825426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:35.825723 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 07:36:35.825959 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 07:36:35.826445 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
I20260926 07:36:35.829286 5269 heartbeater.cc:347] Connected to a master server at 127.22.221.126:40587
I20260926 07:36:35.829644 5269 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:35.830426 5269 heartbeater.cc:511] Master 127.22.221.126:40587 requested a full tablet report, sending...
I20260926 07:36:35.831856 4811 ts_manager.cc:194] Registered new tserver with Master: 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047)
I20260926 07:36:35.832736 4811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:53569
I20260926 07:36:35.842233 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:36:35.859159 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.846714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.22.221.126:40587)
0926 07:36:35.846950 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:35.846958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:35.847339 (+ 381us) server_negotiation.cc:407] Connection header received
0926 07:36:35.847535 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:35.847540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:35.847617 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:35.847721 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:35.849079 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.849816 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.851128 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.851634 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.851779 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.854097 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:35.854103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:35.854108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:35.854150 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:35.856534 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:35.857201 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:35.857208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:35.857211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:35.857347 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:35.857720 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:35.857727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:35.857732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:35.858065 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:35.858322 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:35.858629 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 07:36:35.858854 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:36:35.863128 4811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:36:35.869939 4811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:36:35.912703 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.882284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:46843 (local address 127.0.0.1:54108)
0926 07:36:35.897063 (+ 14779us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:35.897139 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:35.897424 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:35.903369 (+ 5945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:35.903380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:35.903426 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:35.903983 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.904001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.906883 (+ 2882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.906889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.909524 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.909540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.911660 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.911667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.911873 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.911926 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 07:36:35.912030 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:36:35.912223 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14391,"thread_start_us":14194,"threads_started":1}
I20260926 07:36:35.914109 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.882847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.22.221.66:46843)
0926 07:36:35.902252 (+ 19405us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:35.902265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:35.902305 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:36:35.902434 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:35.902446 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:35.902784 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:35.904312 (+ 1528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:35.904570 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.906688 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.909748 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.911388 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.913263 (+ 1875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.913508 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:35.913587 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 07:36:35.913768 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19167,"thread_start_us":20130,"threads_started":1}
I20260926 07:36:35.917037 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.882370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:37047 (local address 127.0.0.1:55142)
0926 07:36:35.908001 (+ 25631us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:35.908021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:35.908147 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:35.909639 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:35.909647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:35.909675 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:35.910168 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.910180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.912344 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.912349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.914162 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.914203 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.916177 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.916184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.916378 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.916412 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:36:35.916523 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:36:35.916660 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25486,"thread_start_us":29085,"threads_started":1}
I20260926 07:36:35.917091 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.882598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.22.221.67:37047)
0926 07:36:35.908723 (+ 26125us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:35.908731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:35.908768 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:36:35.908862 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:35.908870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:35.909147 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:35.909342 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:35.910568 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.912204 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.914400 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.915945 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.916093 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.916344 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:35.916567 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 07:36:35.916726 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25949,"thread_start_us":25798,"threads_started":1}
I20260926 07:36:35.918280 5070 tablet_service.cc:1467] Processing CreateTablet for tablet fd00861f25be41c4bc42383a5c8ae088 (DEFAULT_TABLE table=test-table [id=1332604dd06f4b25b154c4e89bc0fdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:35.919857 5070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd00861f25be41c4bc42383a5c8ae088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:35.920135 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.882767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53890)
0926 07:36:35.911790 (+ 29023us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:35.911806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:35.911951 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:35.912599 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:35.912604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:35.912637 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:35.913034 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.913045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.914995 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.915000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.917152 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:35.917167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.919301 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.919308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:35.919501 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.919532 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:36:35.919630 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:36:35.919756 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28880,"mutex_wait_us":1,"thread_start_us":27544,"threads_started":1}
I20260926 07:36:35.920184 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:35.883270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.22.221.65:39963)
0926 07:36:35.907312 (+ 24042us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:35.907322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:35.911931 (+ 4609us) server_negotiation.cc:407] Connection header received
0926 07:36:35.912109 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:35.912114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:35.912360 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:35.912594 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:35.913268 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.914811 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.917337 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:35.918760 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:35.918880 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:35.919098 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:35.919703 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 07:36:35.919881 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23800,"thread_start_us":24058,"threads_started":1}
I20260926 07:36:35.920706 5204 tablet_service.cc:1467] Processing CreateTablet for tablet fd00861f25be41c4bc42383a5c8ae088 (DEFAULT_TABLE table=test-table [id=1332604dd06f4b25b154c4e89bc0fdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:35.922144 5204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd00861f25be41c4bc42383a5c8ae088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:35.924096 4936 tablet_service.cc:1467] Processing CreateTablet for tablet fd00861f25be41c4bc42383a5c8ae088 (DEFAULT_TABLE table=test-table [id=1332604dd06f4b25b154c4e89bc0fdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:35.925472 4936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd00861f25be41c4bc42383a5c8ae088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:35.933351 5288 tablet_bootstrap.cc:492] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Bootstrap starting.
I20260926 07:36:35.935947 5289 tablet_bootstrap.cc:492] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Bootstrap starting.
I20260926 07:36:35.937001 5288 tablet_bootstrap.cc:654] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:35.938771 5288 log.cc:824] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:35.939566 5289 tablet_bootstrap.cc:654] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:35.939929 5290 tablet_bootstrap.cc:492] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Bootstrap starting.
I20260926 07:36:35.940855 5289 log.cc:824] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:35.942845 5288 tablet_bootstrap.cc:492] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: No bootstrap required, opened a new log
I20260926 07:36:35.943137 5288 ts_tablet_manager.cc:1397] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 07:36:35.943933 5289 tablet_bootstrap.cc:492] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: No bootstrap required, opened a new log
I20260926 07:36:35.944187 5290 tablet_bootstrap.cc:654] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:35.944185 5289 ts_tablet_manager.cc:1397] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:36:35.945294 5290 log.cc:824] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:35.948247 5290 tablet_bootstrap.cc:492] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: No bootstrap required, opened a new log
I20260926 07:36:35.948547 5290 ts_tablet_manager.cc:1397] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:36:35.954460 5288 raft_consensus.cc:348] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:35.954505 5289 raft_consensus.cc:348] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:35.954924 5288 raft_consensus.cc:374] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:35.954926 5289 raft_consensus.cc:374] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:35.955029 5289 raft_consensus.cc:729] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ef4776466046418c871e49e775ca5a, State: Initialized, Role: FOLLOWER
I20260926 07:36:35.955027 5288 raft_consensus.cc:729] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab39e6c0db14929a891d1ac86a5d917, State: Initialized, Role: FOLLOWER
I20260926 07:36:35.955691 5288 consensus_queue.cc:260] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:35.955693 5289 consensus_queue.cc:260] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:35.956965 5269 heartbeater.cc:503] Master 127.22.221.126:40587 was elected leader, sending a full tablet report...
I20260926 07:36:35.957489 5135 heartbeater.cc:503] Master 127.22.221.126:40587 was elected leader, sending a full tablet report...
I20260926 07:36:35.957623 5288 ts_tablet_manager.cc:1428] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 07:36:35.957980 5289 ts_tablet_manager.cc:1428] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 07:36:35.956597 5290 raft_consensus.cc:348] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:35.958266 5290 raft_consensus.cc:374] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:35.958370 5290 raft_consensus.cc:729] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde7fca027b4c41a8be28ed02bf5b65, State: Initialized, Role: FOLLOWER
I20260926 07:36:35.959061 5290 consensus_queue.cc:260] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:35.962131 5290 ts_tablet_manager.cc:1428] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260926 07:36:35.997797 5002 tablet.cc:2369] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:36:36.032008 5136 tablet.cc:2369] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:36:36.057495 5270 tablet.cc:2369] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:36:36.135027 5296 raft_consensus.cc:482] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:36:36.135402 5296 raft_consensus.cc:504] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:36.137421 5296 leader_election.cc:288] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047), 45ef4776466046418c871e49e775ca5a (127.22.221.66:46843)
I20260926 07:36:36.146360 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.138420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:55767 (local address 127.22.221.66:46843)
0926 07:36:36.138603 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:36.138609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:36.140023 (+ 1414us) server_negotiation.cc:407] Connection header received
0926 07:36:36.140184 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:36.140190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:36.140264 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:36.140352 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:36.141003 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.142072 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.144463 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.145407 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.145581 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.145661 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:36.145966 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 07:36:36.146093 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:36:36.146437 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.138020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:56501 (local address 127.22.221.67:37047)
0926 07:36:36.138296 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:36.138306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:36.139562 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 07:36:36.139949 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:36.139954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:36.140060 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:36.140167 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:36.140817 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.141611 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.144268 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.145480 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.145680 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.145755 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:36.146054 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 07:36:36.146183 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:36:36.146539 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.137801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:37047 (local address 127.22.221.65:56501)
0926 07:36:36.139450 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.139471 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:36.139703 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:36.140198 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:36.140203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:36.140225 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:36.140648 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:36.140660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.141781 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.141787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:36.144080 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:36.144096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.145722 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.145729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:36.145890 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.145913 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:36:36.146024 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:36:36.146148 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"mutex_wait_us":75,"thread_start_us":1749,"threads_started":1}
I20260926 07:36:36.146565 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.137861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:46843 (local address 127.22.221.65:55767)
0926 07:36:36.139905 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.139922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:36.140051 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:36.140379 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:36.140383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:36.140413 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:36.140807 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:36.140819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.142224 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.142229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:36.144316 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:36.144326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.145624 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.145631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:36.145804 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.145829 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:36:36.145938 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:36:36.146065 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"thread_start_us":955,"threads_started":1}
I20260926 07:36:36.147722 5224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd00861f25be41c4bc42383a5c8ae088" candidate_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ab39e6c0db14929a891d1ac86a5d917" is_pre_election: true
I20260926 07:36:36.147722 5090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd00861f25be41c4bc42383a5c8ae088" candidate_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ef4776466046418c871e49e775ca5a" is_pre_election: true
I20260926 07:36:36.148319 5224 raft_consensus.cc:2454] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cde7fca027b4c41a8be28ed02bf5b65 in term 0.
I20260926 07:36:36.148303 5090 raft_consensus.cc:2454] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cde7fca027b4c41a8be28ed02bf5b65 in term 0.
I20260926 07:36:36.148993 4890 leader_election.cc:302] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ef4776466046418c871e49e775ca5a, 5cde7fca027b4c41a8be28ed02bf5b65; no voters:
I20260926 07:36:36.149417 5296 raft_consensus.cc:2781] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:36:36.149566 5296 raft_consensus.cc:482] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:36:36.149675 5296 raft_consensus.cc:3037] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:36.151904 5296 raft_consensus.cc:504] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:36.152469 5296 leader_election.cc:288] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [CANDIDATE]: Term 1 election: Requested vote from peers 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047), 45ef4776466046418c871e49e775ca5a (127.22.221.66:46843)
I20260926 07:36:36.152891 5224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd00861f25be41c4bc42383a5c8ae088" candidate_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ab39e6c0db14929a891d1ac86a5d917"
I20260926 07:36:36.152964 5090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd00861f25be41c4bc42383a5c8ae088" candidate_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ef4776466046418c871e49e775ca5a"
I20260926 07:36:36.153082 5224 raft_consensus.cc:3037] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:36.153231 5090 raft_consensus.cc:3037] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:36.155365 5224 raft_consensus.cc:2454] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cde7fca027b4c41a8be28ed02bf5b65 in term 1.
I20260926 07:36:36.155475 5090 raft_consensus.cc:2454] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cde7fca027b4c41a8be28ed02bf5b65 in term 1.
I20260926 07:36:36.155807 4891 leader_election.cc:302] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cde7fca027b4c41a8be28ed02bf5b65, 9ab39e6c0db14929a891d1ac86a5d917; no voters:
I20260926 07:36:36.156128 5296 raft_consensus.cc:2781] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:36.156594 5296 raft_consensus.cc:686] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 LEADER]: Becoming Leader. State: Replica: 5cde7fca027b4c41a8be28ed02bf5b65, State: Running, Role: LEADER
I20260926 07:36:36.157078 5296 consensus_queue.cc:237] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } }
I20260926 07:36:36.161660 4810 catalog_manager.cc:5495] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } health_report { overall_health: HEALTHY } } }
I20260926 07:36:36.209031 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.203473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.22.221.65:39963)
0926 07:36:36.203666 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:36.203670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:36.203688 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:36:36.203951 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:36.203956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:36.204045 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:36.204183 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:36:36.204570 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.205404 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.206953 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.207439 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.207537 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.207990 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:36:36.208520 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:36:36.208634 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:36.208730 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:36:36.208799 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:36:36.217329 5090 raft_consensus.cc:1261] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Refusing update from remote peer 5cde7fca027b4c41a8be28ed02bf5b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:36.217341 5224 raft_consensus.cc:1261] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Refusing update from remote peer 5cde7fca027b4c41a8be28ed02bf5b65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:36.218343 5299 consensus_queue.cc:1035] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:36.218717 5296 consensus_queue.cc:1035] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:36.231053 5302 mvcc.cc:205] Tried to move back new op lower bound from 7333511971696340992 to 7333511971462725632. Current Snapshot: MvccSnapshot[applied={T|T < 7333511971696340992}]
I20260926 07:36:36.242918 4811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:36:36.243188 4811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:36:36.246690 4811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Sending DeleteTablet for 3 replicas of tablet fd00861f25be41c4bc42383a5c8ae088
I20260926 07:36:36.247804 5204 tablet_service.cc:1514] Processing DeleteTablet for tablet fd00861f25be41c4bc42383a5c8ae088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:36 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:55142
I20260926 07:36:36.247928 5070 tablet_service.cc:1514] Processing DeleteTablet for tablet fd00861f25be41c4bc42383a5c8ae088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:36 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:54108
I20260926 07:36:36.248103 4936 tablet_service.cc:1514] Processing DeleteTablet for tablet fd00861f25be41c4bc42383a5c8ae088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:36 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:53890
I20260926 07:36:36.248558 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4873
I20260926 07:36:36.264830 5312 tablet_replica.cc:330] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: stopping tablet replica
I20260926 07:36:36.265446 5312 raft_consensus.cc:2229] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:36:36.266088 5312 raft_consensus.cc:2258] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:36.266407 5314 tablet_replica.cc:330] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: stopping tablet replica
I20260926 07:36:36.266852 5314 raft_consensus.cc:2229] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:36:36.267598 5314 raft_consensus.cc:2258] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:36.268334 5313 tablet_replica.cc:330] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: stopping tablet replica
I20260926 07:36:36.268738 5313 raft_consensus.cc:2229] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:36:36.269238 5313 raft_consensus.cc:2258] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:36.270430 5314 ts_tablet_manager.cc:1905] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:36.271370 5312 ts_tablet_manager.cc:1905] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:36:36 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:36:36.277161 5313 ts_tablet_manager.cc:1905] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:36.278911 5312 ts_tablet_manager.cc:1918] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:36.279078 5312 log.cc:1196] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/fd00861f25be41c4bc42383a5c8ae088
I20260926 07:36:36.279639 5312 ts_tablet_manager.cc:1939] T fd00861f25be41c4bc42383a5c8ae088 P 9ab39e6c0db14929a891d1ac86a5d917: Deleting consensus metadata
I20260926 07:36:36.281642 4798 catalog_manager.cc:4841] TS 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047): tablet fd00861f25be41c4bc42383a5c8ae088 (table test-table [id=1332604dd06f4b25b154c4e89bc0fdf9]) successfully deleted
I20260926 07:36:36.286554 5314 ts_tablet_manager.cc:1918] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:36.289216 5313 ts_tablet_manager.cc:1918] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:36.289397 5313 log.cc:1196] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/fd00861f25be41c4bc42383a5c8ae088
I20260926 07:36:36.288842 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.249917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.22.221.65:39963)
0926 07:36:36.250136 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:36.250145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:36.260812 (+ 10667us) server_negotiation.cc:407] Connection header received
0926 07:36:36.261401 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:36.261411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:36.261521 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:36.261626 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:36.266531 (+ 4905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.267805 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.270423 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.271114 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.271212 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.276992 (+ 5780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:36.277007 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:36.277023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:36.277074 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:36.281312 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:36.283437 (+ 2125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:36.283447 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:36.283452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:36.283635 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:36.286750 (+ 3115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:36.286756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:36.286762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:36.287265 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:36.287553 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:36.288224 (+ 671us) server_negotiation.cc:299] Negotiation successful
0926 07:36:36.288471 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:36:36.289960 5313 ts_tablet_manager.cc:1939] T fd00861f25be41c4bc42383a5c8ae088 P 45ef4776466046418c871e49e775ca5a: Deleting consensus metadata
I20260926 07:36:36.290545 5314 log.cc:1196] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/fd00861f25be41c4bc42383a5c8ae088
I20260926 07:36:36.290714 4996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:36.291940 4797 catalog_manager.cc:4841] TS 45ef4776466046418c871e49e775ca5a (127.22.221.66:46843): tablet fd00861f25be41c4bc42383a5c8ae088 (table test-table [id=1332604dd06f4b25b154c4e89bc0fdf9]) successfully deleted
I20260926 07:36:36.387977 5314 ts_tablet_manager.cc:1939] T fd00861f25be41c4bc42383a5c8ae088 P 5cde7fca027b4c41a8be28ed02bf5b65: Deleting consensus metadata
I20260926 07:36:36.388585 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4873
I20260926 07:36:36.407163 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5007
W20260926 07:36:36.407411 4799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.221.65:39963 (error 108)
I20260926 07:36:36.409349 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.408847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53922)
0926 07:36:36.409040 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.409165 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 07:36:36.409596 4799 catalog_manager.cc:4584] TS 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65:39963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd00861f25be41c4bc42383a5c8ae088: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Sep 26 07:36:36 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:36:36.423017 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.408093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.22.221.66:46843)
0926 07:36:36.408276 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:36.408281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:36.408302 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:36:36.408429 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:36.408439 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:36.408531 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:36.408631 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:36.410145 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.410918 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.412370 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.412752 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.412846 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.416992 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:36.417006 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:36.417021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:36.417060 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:36.420152 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:36.420882 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:36.420890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:36.420893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:36.421013 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:36.421402 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:36.421409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:36.421412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:36.421829 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:36.422128 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:36.422448 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 07:36:36.422667 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:36:36.423746 5130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:36.463373 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.462822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53934)
0926 07:36:36.463055 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.463149 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 07:36:36.464599 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.464166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53950)
0926 07:36:36.464317 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.464420 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:36:36.518497 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5007
I20260926 07:36:36.525415 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.524743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53956)
0926 07:36:36.525036 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.525163 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 07:36:36.526592 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.526159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53966)
0926 07:36:36.526334 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.526434 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:36:36.536306 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5141
Sep 26 07:36:36 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:36:36.551617 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.537036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.22.221.67:37047)
0926 07:36:36.537226 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:36.537234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:36.537258 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:36:36.537489 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:36.537493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:36.537567 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:36.537675 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:36.539012 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.539819 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.541162 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:36.541552 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:36.541694 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:36.546439 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:36.546454 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:36.546470 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:36.546506 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:36.549167 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:36.549818 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:36.549825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:36.549827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:36.549954 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:36.550276 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:36.550279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:36.550281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:36.550632 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:36.550844 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:36.551100 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:36:36.551335 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:36:36.552313 5264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:36.607518 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.606865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53974)
0926 07:36:36.607117 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.607313 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 07:36:36.608769 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:36.608363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53976)
0926 07:36:36.608564 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:36.608641 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 07:36:36.642167 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5141
I20260926 07:36:36.659035 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 4780
I20260926 07:36:36.660275 4842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:36.773096 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 4780
I20260926 07:36:36.794591 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40587
--webserver_interface=127.22.221.126
--webserver_port=34483
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:37.129244 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:37.129530 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:37.129668 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:37.137050 5321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:37.137162 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:37.137274 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:37.137378 5321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:37.137460 5321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:37.147768 5321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=34483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:37.149101 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:37.150419 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:37.173784 5330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:37.173862 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:37.173818 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:37.174986 5321 server_base.cc:1034] running on GCE node
I20260926 07:36:37.175921 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:37.177590 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:37.178797 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790408197178732 us; error 72 us; skew 500 ppm
I20260926 07:36:37.179279 5321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:37 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408197, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:37.182848 5321 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:37.184556 5321 webserver.cc:460] Webserver started at http://127.22.221.126:34483/ using document root <none> and password file <none>
I20260926 07:36:37.185333 5321 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:37.185490 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:37.191658 5321 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:36:37.194694 5337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:37.196316 5321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:37.196537 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "25ae8838b3ab40e2b2fca33613706e54"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "ec5e72cfc57983eba0576870fb3abd34"
server_key_iv: "67e206faa3a6a14bd14c9bc7cc042e45"
server_key_version: "encryptionkey@0"
I20260926 07:36:37.197729 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:37.222882 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:37.227555 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:37.227864 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:37.247217 5321 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:40587
I20260926 07:36:37.247291 5388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:40587 every 8 connection(s)
I20260926 07:36:37.248850 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:37.256017 5389 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:36:37.257596 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5321
I20260926 07:36:37.258407 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:39963
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:40587
--webserver_port=43283
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:37.259433 5389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap starting.
I20260926 07:36:37.279932 5389 log.cc:824] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:37.292104 5389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:36:37.292650 5389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap complete.
I20260926 07:36:37.301736 5389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:37.302635 5389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae8838b3ab40e2b2fca33613706e54, State: Initialized, Role: FOLLOWER
I20260926 07:36:37.303372 5389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:37.303622 5389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:37.303740 5389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:37.303949 5389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:36:37.307756 5389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:37.308311 5389 leader_election.cc:302] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ae8838b3ab40e2b2fca33613706e54; no voters:
I20260926 07:36:37.308990 5389 leader_election.cc:288] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:36:37.309125 5393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:36:37.310443 5393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 LEADER]: Becoming Leader. State: Replica: 25ae8838b3ab40e2b2fca33613706e54, State: Running, Role: LEADER
I20260926 07:36:37.310992 5393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:37.311470 5389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:37.313817 5394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "25ae8838b3ab40e2b2fca33613706e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } } }
I20260926 07:36:37.314177 5394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:37.314918 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ae8838b3ab40e2b2fca33613706e54. Latest consensus state: current_term: 2 leader_uuid: "25ae8838b3ab40e2b2fca33613706e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } } }
I20260926 07:36:37.315323 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:37.317629 5398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:37.327533 5398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:37.329430 5398 catalog_manager.cc:1259] Loaded cluster ID: aee55d59ebeb418f8ec94476b8001a48
I20260926 07:36:37.329560 5398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:37.334438 5398 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 07:36:37.356169 5398 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 07:36:37.363107 5398 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:37.364491 5398 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Loaded TSK: 0
I20260926 07:36:37.365720 5398 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:36:37.630510 5391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:37.630831 5391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:37.630992 5391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:37.638319 5391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:37.638468 5391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:37.638655 5391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:36:37.648712 5391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:39963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=43283
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:37.650033 5391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:37.651463 5391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:37.666409 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:37.666448 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:37.667006 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:37.667297 5391 server_base.cc:1034] running on GCE node
I20260926 07:36:37.668015 5391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:37.669023 5391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:37.670267 5391 hybrid_clock.cc:648] HybridClock initialized: now 1790408197670197 us; error 63 us; skew 500 ppm
I20260926 07:36:37.670600 5391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:37 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408197, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:37.679718 5391 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:36:37.681469 5391 webserver.cc:460] Webserver started at http://127.22.221.65:43283/ using document root <none> and password file <none>
I20260926 07:36:37.682199 5391 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:37.682360 5391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:37.687867 5391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:37.690650 5426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:37.692152 5391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:36:37.692312 5391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "5cde7fca027b4c41a8be28ed02bf5b65"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "8b9bbdfad55107964a7212a375d4c24b"
server_key_iv: "d972e4f3e8d63705ff1ab1b1808234cb"
server_key_version: "encryptionkey@0"
I20260926 07:36:37.693260 5391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:37.715366 5391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:37.719162 5391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:37.719502 5391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:37.720574 5391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:37.721959 5391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:37.722054 5391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:37.722208 5391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:37.722301 5391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:37.754470 5391 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:39963
I20260926 07:36:37.754568 5538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:39963 every 8 connection(s)
I20260926 07:36:37.755944 5391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:36:37.762699 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5391
I20260926 07:36:37.763428 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:46843
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:40587
--webserver_port=36463
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:36:37 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408197, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:37.781436 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:37.758970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:39225 (local address 127.22.221.126:40587)
0926 07:36:37.759700 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:37.759720 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:37.760224 (+ 504us) server_negotiation.cc:407] Connection header received
0926 07:36:37.762425 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:37.762439 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:37.762874 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:37.763334 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:37.764957 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:37.766224 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:37.767923 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:37.768640 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:37.768858 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:37.773434 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:37.773443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:37.773461 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:37.773503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:37.776589 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:37.777645 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:37.777652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:37.777659 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:37.777832 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:37.778366 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:37.778369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:37.778371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:37.778924 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:37.779078 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:37.780076 (+ 998us) server_negotiation.cc:299] Negotiation successful
0926 07:36:37.780399 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":299,"threads_started":1}
I20260926 07:36:37.782001 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:37.759198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:39225)
0926 07:36:37.760038 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:37.760075 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:37.761816 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:37.763478 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:37.763491 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:37.764127 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:37.764681 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:37.764699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:37.766420 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:37.766425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:37.767764 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:37.767778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:37.768907 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:37.768915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:37.769188 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:37.769929 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:37.769967 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:37.773063 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:37.776787 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:37.776799 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:37.776820 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:37.777485 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:37.777971 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:37.777978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:37.777985 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:37.778242 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:37.779073 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:37.779084 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:37.779570 (+ 486us) client_negotiation.cc:769] Sending connection context
0926 07:36:37.779900 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 07:36:37.780524 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":331,"threads_started":1}
I20260926 07:36:37.784880 5539 heartbeater.cc:347] Connected to a master server at 127.22.221.126:40587
I20260926 07:36:37.785223 5539 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:37.785948 5539 heartbeater.cc:511] Master 127.22.221.126:40587 requested a full tablet report, sending...
I20260926 07:36:37.788209 5354 ts_manager.cc:194] Registered new tserver with Master: 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65:39963)
I20260926 07:36:37.789645 5354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:39225
W20260926 07:36:38.108153 5543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:38.108486 5543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:38.108667 5543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:38.116235 5543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:38.116397 5543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:38.116592 5543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:36:38.126809 5543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:46843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=36463
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:38.128224 5543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:38.129573 5543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:38.143057 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:38.143200 5548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:38.143185 5551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:38.143836 5543 server_base.cc:1034] running on GCE node
I20260926 07:36:38.144482 5543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:38.145501 5543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:38.146733 5543 hybrid_clock.cc:648] HybridClock initialized: now 1790408198146677 us; error 53 us; skew 500 ppm
I20260926 07:36:38.147051 5543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:38 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408198, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:38.150164 5543 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:36:38.151664 5543 webserver.cc:460] Webserver started at http://127.22.221.66:36463/ using document root <none> and password file <none>
I20260926 07:36:38.152300 5543 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:38.152416 5543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:38.157678 5543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 07:36:38.160676 5558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:38.162009 5543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:36:38.162173 5543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "45ef4776466046418c871e49e775ca5a"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "14df5bea1539a0d5048bf6bf0f4c6232"
server_key_iv: "41add901c5e1d87f76689b0ad271c8ba"
server_key_version: "encryptionkey@0"
I20260926 07:36:38.163130 5543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:38.188182 5543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:38.192600 5543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:38.192896 5543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:38.193919 5543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:38.195221 5543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:38.195400 5543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:38.195503 5543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:38.195602 5543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:38.230531 5543 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:46843
I20260926 07:36:38.230618 5670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:46843 every 8 connection(s)
I20260926 07:36:38.232041 5543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:36:38.240144 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5543
I20260926 07:36:38.240772 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:37047
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:40587
--webserver_port=33601
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:36:38 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408198, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:38.256526 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.234535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:48263 (local address 127.22.221.126:40587)
0926 07:36:38.234779 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.234783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.236098 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 07:36:38.237919 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.237923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.237988 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.238082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:38.239641 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.240380 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.242010 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.242466 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.242604 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.248584 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:38.248593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:38.248600 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:38.248651 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:38.252624 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.253637 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.253645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.253651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.253807 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.254313 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.254321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.254327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.254726 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:38.254880 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.255749 (+ 869us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.256043 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 07:36:38.257422 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.235060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:48263)
0926 07:36:38.235942 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.235980 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.237677 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.238319 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.238327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.238833 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:38.239477 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.239497 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.240530 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.240542 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.241860 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.241873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.242690 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.242696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.242934 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.243653 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:38.243683 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:38.248328 (+ 4645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:38.252792 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:38.252810 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:38.252831 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:38.253471 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:38.253937 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:38.253942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:38.253946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:38.254171 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:38.254865 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:38.254873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:38.255302 (+ 429us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.255600 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.256165 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":405,"threads_started":1}
I20260926 07:36:38.259022 5671 heartbeater.cc:347] Connected to a master server at 127.22.221.126:40587
I20260926 07:36:38.259409 5671 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:38.260274 5671 heartbeater.cc:511] Master 127.22.221.126:40587 requested a full tablet report, sending...
I20260926 07:36:38.261967 5354 ts_manager.cc:194] Registered new tserver with Master: 45ef4776466046418c871e49e775ca5a (127.22.221.66:46843)
I20260926 07:36:38.263435 5354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:48263
W20260926 07:36:38.575491 5674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:38.575771 5674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:38.575964 5674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:38.583109 5674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:38.583250 5674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:38.583467 5674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:36:38.593129 5674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:37047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=33601
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:38.594439 5674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:38.595808 5674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:38.609596 5679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:38.609541 5680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:38.609575 5682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:38.610831 5674 server_base.cc:1034] running on GCE node
I20260926 07:36:38.611632 5674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:38.612614 5674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:38.613824 5674 hybrid_clock.cc:648] HybridClock initialized: now 1790408198613760 us; error 69 us; skew 500 ppm
I20260926 07:36:38.614163 5674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:38 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408198, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:38.617347 5674 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:36:38.618757 5674 webserver.cc:460] Webserver started at http://127.22.221.67:33601/ using document root <none> and password file <none>
I20260926 07:36:38.619524 5674 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:38.619671 5674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:38.624852 5674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:36:38.627717 5689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:38.629134 5674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:36:38.629287 5674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "9ab39e6c0db14929a891d1ac86a5d917"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "8410112c2bb075212d02bcb4baf48dcb"
server_key_iv: "f6e1f068b5d8672e2f334735e42e3f91"
server_key_version: "encryptionkey@0"
I20260926 07:36:38.630206 5674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:38.647456 5674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:38.651495 5674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:38.651737 5674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:38.652702 5674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:38.654011 5674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:38.654105 5674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:38.654181 5674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:38.654237 5674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:38.688555 5674 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:37047
I20260926 07:36:38.688603 5801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:37047 every 8 connection(s)
I20260926 07:36:38.689970 5674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:36:38.699395 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5674
Sep 26 07:36:38 dist-test-slave-qlq8 krb5kdc[4767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408198, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:38.712181 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.692682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:58725 (local address 127.22.221.126:40587)
0926 07:36:38.692866 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.692871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.693984 (+ 1113us) server_negotiation.cc:407] Connection header received
0926 07:36:38.695824 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.695829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.695901 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.695999 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:38.697852 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.698596 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.700357 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.701025 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.701180 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.704896 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:38.704903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:38.704908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:38.704948 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:38.708500 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.709318 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.709324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.709327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.709493 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.710019 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.710023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.710025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.710349 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:38.710475 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.711463 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.711787 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:36:38.713431 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.693161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:58725)
0926 07:36:38.693830 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.693873 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.695517 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.696390 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.696398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.696969 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:38.697664 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.697682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.698767 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.698772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.700203 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.700219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.701258 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.701266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.701584 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.702313 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:38.702334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:38.704677 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:38.708684 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:38.708692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:38.708712 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:38.709168 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:38.709610 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:38.709613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:38.709616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:38.709892 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:38.710487 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:38.710498 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:38.710979 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.711326 (+ 347us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.711972 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":263,"threads_started":1}
I20260926 07:36:38.715050 5802 heartbeater.cc:347] Connected to a master server at 127.22.221.126:40587
I20260926 07:36:38.715411 5802 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:38.716127 5802 heartbeater.cc:511] Master 127.22.221.126:40587 requested a full tablet report, sending...
I20260926 07:36:38.716665 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.700339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.22.221.126:40587)
0926 07:36:38.701206 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.701213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.701238 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:36:38.701289 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.701293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.701367 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.701481 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:38.703210 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.704104 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.706009 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.706468 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.706569 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.709953 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:38.709963 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:38.709968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:38.710014 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:38.713701 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.714548 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.714553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.714556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.714695 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.715070 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.715076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.715079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.715474 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:38.715712 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.716034 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.716256 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":632,"threads_started":1}
I20260926 07:36:38.717805 5354 ts_manager.cc:194] Registered new tserver with Master: 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047)
I20260926 07:36:38.718951 5354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:58725
I20260926 07:36:38.719419 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:36:38.726516 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.720489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.22.221.126:40587)
0926 07:36:38.720663 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.720668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.721164 (+ 496us) server_negotiation.cc:407] Connection header received
0926 07:36:38.721446 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.721450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.721568 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.721715 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:36:38.722128 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.722812 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.724402 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.724827 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.724950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.725466 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:36:38.725993 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:36:38.726142 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.726207 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.726281 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:36:38.727437 5354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:36:38.735025 5354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:36:38.771420 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.747976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.22.221.66:46843)
0926 07:36:38.759297 (+ 11321us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.759310 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.759369 (+ 59us) server_negotiation.cc:407] Connection header received
0926 07:36:38.759511 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.759522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.759828 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.761248 (+ 1420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:38.761540 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.763437 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.768181 (+ 4744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.769608 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.769744 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.769978 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.770904 (+ 926us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.771080 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11082,"thread_start_us":12040,"threads_started":1}
I20260926 07:36:38.771556 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.747622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:46843 (local address 127.0.0.1:54124)
0926 07:36:38.748428 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.748469 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.748629 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.760375 (+ 11746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.760388 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.760433 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:38.760970 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.760986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.764674 (+ 3688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.764682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.767985 (+ 3303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.768005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.770167 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.770173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.770391 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.770566 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.770827 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.771041 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":44,"thread_start_us":412,"threads_started":1}
I20260926 07:36:38.774942 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.747933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:53990)
0926 07:36:38.764263 (+ 16330us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.764290 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.764458 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.766331 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.766336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.766368 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:38.767020 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.767035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.769285 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.769290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.771902 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.771921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.774063 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.774070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.774282 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.774321 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.774422 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.774552 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16100,"mutex_wait_us":33,"thread_start_us":15783,"threads_started":1}
I20260926 07:36:38.775372 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.748620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.22.221.65:39963)
0926 07:36:38.763180 (+ 14560us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.763189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.765474 (+ 2285us) server_negotiation.cc:407] Connection header received
0926 07:36:38.765587 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.765596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.765851 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.766052 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:38.767280 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.769074 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.772126 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.773807 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.773963 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.774200 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.774512 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.774805 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14375,"thread_start_us":17710,"threads_started":1}
I20260926 07:36:38.775480 5606 tablet_service.cc:1467] Processing CreateTablet for tablet bfcb7478b7cb487f9465b74e4c01a685 (DEFAULT_TABLE table=test-table [id=8e36325b980b411da03226a3cf728c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:38.777097 5606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcb7478b7cb487f9465b74e4c01a685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:38.778259 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.749468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:37047 (local address 127.0.0.1:55166)
0926 07:36:38.768181 (+ 18713us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.768201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.768334 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.768976 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.768981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.769011 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:38.769435 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.769447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.772126 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.772132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.774355 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.774368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.777310 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.777317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.777520 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.777557 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.777704 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.777856 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18574,"thread_start_us":18338,"threads_started":1}
I20260926 07:36:38.778272 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.750167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.22.221.67:37047)
0926 07:36:38.767523 (+ 17356us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.767535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.768330 (+ 795us) server_negotiation.cc:407] Connection header received
0926 07:36:38.768492 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.768501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.768758 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.768961 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:38.769689 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.771989 (+ 2300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.774599 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.777055 (+ 2456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.777244 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.777487 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.777760 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.777924 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17160,"thread_start_us":16975,"threads_started":1}
I20260926 07:36:38.779364 5474 tablet_service.cc:1467] Processing CreateTablet for tablet bfcb7478b7cb487f9465b74e4c01a685 (DEFAULT_TABLE table=test-table [id=8e36325b980b411da03226a3cf728c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:38.780792 5474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcb7478b7cb487f9465b74e4c01a685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:38.781566 5737 tablet_service.cc:1467] Processing CreateTablet for tablet bfcb7478b7cb487f9465b74e4c01a685 (DEFAULT_TABLE table=test-table [id=8e36325b980b411da03226a3cf728c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:36:38.782704 5737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcb7478b7cb487f9465b74e4c01a685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:38.792897 5539 heartbeater.cc:503] Master 127.22.221.126:40587 was elected leader, sending a full tablet report...
I20260926 07:36:38.795217 5818 tablet_bootstrap.cc:492] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Bootstrap starting.
I20260926 07:36:38.797444 5819 tablet_bootstrap.cc:492] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Bootstrap starting.
I20260926 07:36:38.798444 5820 tablet_bootstrap.cc:492] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Bootstrap starting.
I20260926 07:36:38.799727 5818 tablet_bootstrap.cc:654] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:38.800701 5818 log.cc:824] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:38.802201 5820 tablet_bootstrap.cc:654] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:38.802834 5819 tablet_bootstrap.cc:654] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:38.803653 5820 log.cc:824] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:38.804390 5819 log.cc:824] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:38.804644 5818 tablet_bootstrap.cc:492] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: No bootstrap required, opened a new log
I20260926 07:36:38.804935 5818 ts_tablet_manager.cc:1397] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 07:36:38.807440 5820 tablet_bootstrap.cc:492] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: No bootstrap required, opened a new log
I20260926 07:36:38.807701 5820 ts_tablet_manager.cc:1397] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260926 07:36:38.809976 5819 tablet_bootstrap.cc:492] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: No bootstrap required, opened a new log
I20260926 07:36:38.810262 5819 ts_tablet_manager.cc:1397] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260926 07:36:38.812536 5818 raft_consensus.cc:348] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.812912 5818 raft_consensus.cc:374] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:38.813022 5818 raft_consensus.cc:729] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ef4776466046418c871e49e775ca5a, State: Initialized, Role: FOLLOWER
I20260926 07:36:38.813668 5818 consensus_queue.cc:260] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.814810 5671 heartbeater.cc:503] Master 127.22.221.126:40587 was elected leader, sending a full tablet report...
I20260926 07:36:38.816229 5818 ts_tablet_manager.cc:1428] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 07:36:38.819485 5820 raft_consensus.cc:348] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.819975 5820 raft_consensus.cc:374] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:38.820082 5820 raft_consensus.cc:729] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde7fca027b4c41a8be28ed02bf5b65, State: Initialized, Role: FOLLOWER
I20260926 07:36:38.820773 5820 consensus_queue.cc:260] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.822208 5819 raft_consensus.cc:348] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.822705 5819 raft_consensus.cc:374] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:38.822765 5820 ts_tablet_manager.cc:1428] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 07:36:38.822885 5819 raft_consensus.cc:729] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab39e6c0db14929a891d1ac86a5d917, State: Initialized, Role: FOLLOWER
I20260926 07:36:38.823683 5819 consensus_queue.cc:260] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.824865 5802 heartbeater.cc:503] Master 127.22.221.126:40587 was elected leader, sending a full tablet report...
I20260926 07:36:38.825675 5819 ts_tablet_manager.cc:1428] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 07:36:38.838562 5824 raft_consensus.cc:482] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:36:38.838801 5824 raft_consensus.cc:504] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.840404 5824 leader_election.cc:288] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65:39963), 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047)
I20260926 07:36:38.849769 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.840952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:52899 (local address 127.22.221.65:39963)
0926 07:36:38.841154 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.841160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.841972 (+ 812us) server_negotiation.cc:407] Connection header received
0926 07:36:38.842176 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.842181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.842256 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.842370 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:38.843149 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.844295 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.847319 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.848545 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.848707 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.848781 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.849400 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.849519 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:36:38.849764 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.840999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:41661 (local address 127.22.221.67:37047)
0926 07:36:38.841180 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.841186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.841972 (+ 786us) server_negotiation.cc:407] Connection header received
0926 07:36:38.842175 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.842180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.842256 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.842356 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:36:38.843149 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.844321 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.847317 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.848547 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.848707 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.848781 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.849389 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.849518 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:36:38.850098 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.840659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.22.221.66:52899)
0926 07:36:38.841824 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.841851 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.842056 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.842391 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.842397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.842445 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:38.843018 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.843028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.844469 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.844476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.847134 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.847150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.848820 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.848825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.849086 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.849247 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.849346 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.849544 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":312,"spinlock_wait_cycles":34944,"thread_start_us":746,"threads_started":1}
I20260926 07:36:38.850250 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.840766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:37047 (local address 127.22.221.66:41661)
0926 07:36:38.841859 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:38.841878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:38.842011 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:38.842391 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:38.842397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:38.842451 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:36:38.842952 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.842972 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.844469 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.844476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.847138 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:38.847150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.848801 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.848813 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:38.849084 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.849247 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 07:36:38.849346 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:36:38.849544 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":280,"spinlock_wait_cycles":29184,"thread_start_us":588,"threads_started":1}
I20260926 07:36:38.851049 5494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcb7478b7cb487f9465b74e4c01a685" candidate_uuid: "45ef4776466046418c871e49e775ca5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" is_pre_election: true
I20260926 07:36:38.851166 5757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcb7478b7cb487f9465b74e4c01a685" candidate_uuid: "45ef4776466046418c871e49e775ca5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ab39e6c0db14929a891d1ac86a5d917" is_pre_election: true
I20260926 07:36:38.851627 5494 raft_consensus.cc:2454] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ef4776466046418c871e49e775ca5a in term 0.
I20260926 07:36:38.851691 5757 raft_consensus.cc:2454] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ef4776466046418c871e49e775ca5a in term 0.
I20260926 07:36:38.852458 5562 leader_election.cc:302] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45ef4776466046418c871e49e775ca5a, 5cde7fca027b4c41a8be28ed02bf5b65, 9ab39e6c0db14929a891d1ac86a5d917; no voters:
I20260926 07:36:38.852913 5824 raft_consensus.cc:2781] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:36:38.853057 5824 raft_consensus.cc:482] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:36:38.853157 5824 raft_consensus.cc:3037] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:38.855392 5824 raft_consensus.cc:504] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.856006 5824 leader_election.cc:288] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [CANDIDATE]: Term 1 election: Requested vote from peers 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65:39963), 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047)
I20260926 07:36:38.856364 5494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcb7478b7cb487f9465b74e4c01a685" candidate_uuid: "45ef4776466046418c871e49e775ca5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cde7fca027b4c41a8be28ed02bf5b65"
I20260926 07:36:38.856494 5757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcb7478b7cb487f9465b74e4c01a685" candidate_uuid: "45ef4776466046418c871e49e775ca5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ab39e6c0db14929a891d1ac86a5d917"
I20260926 07:36:38.856568 5494 raft_consensus.cc:3037] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:38.856719 5757 raft_consensus.cc:3037] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:38.858767 5494 raft_consensus.cc:2454] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ef4776466046418c871e49e775ca5a in term 1.
I20260926 07:36:38.858956 5757 raft_consensus.cc:2454] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ef4776466046418c871e49e775ca5a in term 1.
I20260926 07:36:38.859184 5562 leader_election.cc:302] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ef4776466046418c871e49e775ca5a, 5cde7fca027b4c41a8be28ed02bf5b65; no voters:
I20260926 07:36:38.859493 5824 raft_consensus.cc:2781] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:38.859949 5824 raft_consensus.cc:686] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 1 LEADER]: Becoming Leader. State: Replica: 45ef4776466046418c871e49e775ca5a, State: Running, Role: LEADER
I20260926 07:36:38.860414 5824 consensus_queue.cc:237] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } }
I20260926 07:36:38.864928 5354 catalog_manager.cc:5495] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 45ef4776466046418c871e49e775ca5a (127.22.221.66). New cstate: current_term: 1 leader_uuid: "45ef4776466046418c871e49e775ca5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ef4776466046418c871e49e775ca5a" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20260926 07:36:38.889276 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.884197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.22.221.66:46843)
0926 07:36:38.884382 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.884387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.884467 (+ 80us) server_negotiation.cc:407] Connection header received
0926 07:36:38.884632 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.884636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.884709 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.884832 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:36:38.885164 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.885873 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.887219 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.887694 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.887786 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.888279 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:36:38.888772 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:36:38.888888 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.888998 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.889064 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:36:38.897884 5757 raft_consensus.cc:1261] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Refusing update from remote peer 45ef4776466046418c871e49e775ca5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:38.897907 5494 raft_consensus.cc:1261] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Refusing update from remote peer 45ef4776466046418c871e49e775ca5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:36:38.898931 5829 consensus_queue.cc:1035] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ab39e6c0db14929a891d1ac86a5d917" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:38.899296 5824 consensus_queue.cc:1035] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cde7fca027b4c41a8be28ed02bf5b65" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:36:38.928406 5353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:36:38.928730 5353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:36:38.932757 5353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Sending DeleteTablet for 3 replicas of tablet bfcb7478b7cb487f9465b74e4c01a685
I20260926 07:36:38.934041 5606 tablet_service.cc:1514] Processing DeleteTablet for tablet bfcb7478b7cb487f9465b74e4c01a685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:38 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:54124
I20260926 07:36:38.934093 5737 tablet_service.cc:1514] Processing DeleteTablet for tablet bfcb7478b7cb487f9465b74e4c01a685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:38 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:55166
I20260926 07:36:38.934238 5474 tablet_service.cc:1514] Processing DeleteTablet for tablet bfcb7478b7cb487f9465b74e4c01a685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:36:38 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:53990
I20260926 07:36:38.935729 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5391
I20260926 07:36:38.947819 5844 tablet_replica.cc:330] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: stopping tablet replica
I20260926 07:36:38.948433 5844 raft_consensus.cc:2229] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:36:38.949184 5844 raft_consensus.cc:2258] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:38.952301 5844 ts_tablet_manager.cc:1905] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:38.961961 5844 ts_tablet_manager.cc:1918] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:38.963804 5843 tablet_replica.cc:330] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: stopping tablet replica
I20260926 07:36:38.963976 5842 tablet_replica.cc:330] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: stopping tablet replica
I20260926 07:36:38.963871 5844 log.cc:1196] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/bfcb7478b7cb487f9465b74e4c01a685
I20260926 07:36:38.964305 5843 raft_consensus.cc:2229] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:36:38.963708 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:38.937011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.22.221.65:39963)
0926 07:36:38.937222 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:38.937228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:38.937246 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:36:38.937303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:38.937308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:38.937393 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:38.937494 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:38.939540 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.940759 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.944215 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:38.944923 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:38.945053 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:38.949065 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:38.949086 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:38.949110 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:38.949169 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:38.954348 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.960588 (+ 6240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.960602 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.960608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.960842 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:38.961287 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:38.961294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:38.961300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:38.961904 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:38.962297 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:38.962594 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 07:36:38.962926 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:36:38.964425 5842 raft_consensus.cc:2229] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:36:38.964941 5842 raft_consensus.cc:2258] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:38.965111 5843 raft_consensus.cc:2258] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:36:38.965315 5534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:38.967478 5842 ts_tablet_manager.cc:1905] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:38.967839 5843 ts_tablet_manager.cc:1905] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:36:38.975441 5842 ts_tablet_manager.cc:1918] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:38.975674 5842 log.cc:1196] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/bfcb7478b7cb487f9465b74e4c01a685
I20260926 07:36:38.976284 5842 ts_tablet_manager.cc:1939] T bfcb7478b7cb487f9465b74e4c01a685 P 9ab39e6c0db14929a891d1ac86a5d917: Deleting consensus metadata
I20260926 07:36:38.978258 5340 catalog_manager.cc:4841] TS 9ab39e6c0db14929a891d1ac86a5d917 (127.22.221.67:37047): tablet bfcb7478b7cb487f9465b74e4c01a685 (table test-table [id=8e36325b980b411da03226a3cf728c09]) successfully deleted
I20260926 07:36:38.980046 5843 ts_tablet_manager.cc:1918] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:36:38.980237 5843 log.cc:1196] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/bfcb7478b7cb487f9465b74e4c01a685
I20260926 07:36:38.980832 5843 ts_tablet_manager.cc:1939] T bfcb7478b7cb487f9465b74e4c01a685 P 45ef4776466046418c871e49e775ca5a: Deleting consensus metadata
I20260926 07:36:38.982728 5339 catalog_manager.cc:4841] TS 45ef4776466046418c871e49e775ca5a (127.22.221.66:46843): tablet bfcb7478b7cb487f9465b74e4c01a685 (table test-table [id=8e36325b980b411da03226a3cf728c09]) successfully deleted
I20260926 07:36:39.067804 5844 ts_tablet_manager.cc:1939] T bfcb7478b7cb487f9465b74e4c01a685 P 5cde7fca027b4c41a8be28ed02bf5b65: Deleting consensus metadata
I20260926 07:36:39.068462 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5391
W20260926 07:36:39.087047 5341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.221.65:39963 (error 108)
I20260926 07:36:39.087383 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5543
I20260926 07:36:39.089573 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.089052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54012)
0926 07:36:39.089261 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.089389 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:36:39.089857 5341 catalog_manager.cc:4584] TS 5cde7fca027b4c41a8be28ed02bf5b65 (127.22.221.65:39963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfcb7478b7cb487f9465b74e4c01a685: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
I20260926 07:36:39.103783 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.088223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.22.221.66:46843)
0926 07:36:39.088424 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:39.088430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:39.088519 (+ 89us) server_negotiation.cc:407] Connection header received
0926 07:36:39.088677 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:39.088683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:39.088785 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:39.088910 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:39.090478 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:39.091728 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:39.093088 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:39.093788 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:39.093928 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:39.096640 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:39.096656 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:39.096680 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:39.096725 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:39.100804 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:39.101561 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:39.101570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:39.101574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:39.101734 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:39.102114 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:39.102120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:39.102124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:39.102630 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:39.102937 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:39.103119 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:36:39.103395 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:36:39.104507 5666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:39.143886 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.143257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54016)
0926 07:36:39.143490 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.143678 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 07:36:39.145056 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.144695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54030)
0926 07:36:39.144847 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.144936 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:36:39.205711 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.205242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54046)
0926 07:36:39.205456 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.205546 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:36:39.206760 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.206427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54062)
0926 07:36:39.206554 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.206625 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:36:39.222546 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5543
I20260926 07:36:39.241830 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5674
I20260926 07:36:39.261718 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.247200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.22.221.67:37047)
0926 07:36:39.247758 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:39.247764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:39.247793 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:36:39.247854 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:39.247859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:39.247932 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:39.248024 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:39.249446 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:39.250517 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:39.251973 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:39.252607 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:39.252732 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:39.255260 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:39.255280 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:39.255302 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:39.255346 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:39.258864 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:39.259608 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:39.259619 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:39.259623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:39.259806 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:39.260157 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:39.260162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:39.260166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:39.260600 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:39.260896 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:39.261140 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 07:36:39.261368 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430}
I20260926 07:36:39.262449 5797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:39.287165 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.286721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54070)
0926 07:36:39.286929 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.287014 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:36:39.288254 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:39.287861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:39963 (local address 127.0.0.1:54086)
0926 07:36:39.288050 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:39.288126 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 07:36:39.353138 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5674
I20260926 07:36:39.369709 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5321
I20260926 07:36:39.370806 5384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:39.484549 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5321
I20260926 07:36:39.506058 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40587
--webserver_interface=127.22.221.126
--webserver_port=34483
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:39.841284 5850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:39.841584 5850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:39.841771 5850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:39.849311 5850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:39.849411 5850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:39.849510 5850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:39.849601 5850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:39.849676 5850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:39.859783 5850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=34483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:39.861071 5850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:39.862409 5850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:39.874312 5855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:39.874385 5856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:39.874413 5858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:39.875204 5850 server_base.cc:1034] running on GCE node
I20260926 07:36:39.876006 5850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:39.877359 5850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:39.878607 5850 hybrid_clock.cc:648] HybridClock initialized: now 1790408199878571 us; error 54 us; skew 500 ppm
I20260926 07:36:39.878965 5850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:39 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408199, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:39.882035 5850 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:39.883564 5850 webserver.cc:460] Webserver started at http://127.22.221.126:34483/ using document root <none> and password file <none>
I20260926 07:36:39.884230 5850 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:39.884379 5850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:39.889605 5850 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 07:36:39.892429 5865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:39.893895 5850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:36:39.894110 5850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "25ae8838b3ab40e2b2fca33613706e54"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "ec5e72cfc57983eba0576870fb3abd34"
server_key_iv: "67e206faa3a6a14bd14c9bc7cc042e45"
server_key_version: "encryptionkey@0"
I20260926 07:36:39.895131 5850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:39.913069 5850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:39.917043 5850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:39.917326 5850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:39.937011 5850 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:40587
I20260926 07:36:39.937203 5916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:40587 every 8 connection(s)
I20260926 07:36:39.938637 5850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:39.941152 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5850
I20260926 07:36:39.941994 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:39963
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:40587
--webserver_port=43283
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:39.946491 5917 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:36:39.950915 5917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap starting.
I20260926 07:36:39.970969 5917 log.cc:824] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:39.987759 5917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:36:39.988313 5917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54: Bootstrap complete.
I20260926 07:36:39.997491 5917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:39.998467 5917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae8838b3ab40e2b2fca33613706e54, State: Initialized, Role: FOLLOWER
I20260926 07:36:39.999150 5917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:39.999501 5917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:39.999630 5917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:39.999847 5917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:36:40.003702 5917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:40.004196 5917 leader_election.cc:302] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ae8838b3ab40e2b2fca33613706e54; no voters:
I20260926 07:36:40.004885 5917 leader_election.cc:288] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:36:40.005024 5921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:36:40.006215 5921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [term 3 LEADER]: Becoming Leader. State: Replica: 25ae8838b3ab40e2b2fca33613706e54, State: Running, Role: LEADER
I20260926 07:36:40.006816 5921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } }
I20260926 07:36:40.007050 5917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:40.009173 5923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ae8838b3ab40e2b2fca33613706e54. Latest consensus state: current_term: 3 leader_uuid: "25ae8838b3ab40e2b2fca33613706e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } } }
I20260926 07:36:40.009541 5923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:40.010098 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "25ae8838b3ab40e2b2fca33613706e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae8838b3ab40e2b2fca33613706e54" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 40587 } } }
I20260926 07:36:40.010382 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae8838b3ab40e2b2fca33613706e54 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:40.014493 5927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:40.023626 5927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:40.025300 5927 catalog_manager.cc:1259] Loaded cluster ID: aee55d59ebeb418f8ec94476b8001a48
I20260926 07:36:40.025496 5927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 07:36:40.027751 5927 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f115b1ed9ad google::LogMessage::Fail() at ??:0
@ 0x7f115b1f088a google::LogMessage::SendToLog() at ??:0
@ 0x7f115b1ed4de google::LogMessage::Flush() at ??:0
@ 0x7f115b1eea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1170c20512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1170c1d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1170ca5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1170ca5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f115de39345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f115de3c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f115de3c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f115de17eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f11610f16db start_thread at ??:0
@ 0x7f115a08761f clone at ??:0
W20260926 07:36:40.393271 5919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:40.393541 5919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:40.393637 5919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:40.400926 5919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:40.401041 5919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:40.401160 5919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:36:40.410849 5919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:39963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=43283
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:40.412151 5919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:40.413414 5919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:40.426848 5959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:40.427033 5958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:40.426903 5961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:40.427754 5919 server_base.cc:1034] running on GCE node
I20260926 07:36:40.428392 5919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:40.429373 5919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:40.430647 5919 hybrid_clock.cc:648] HybridClock initialized: now 1790408200430581 us; error 57 us; skew 500 ppm
I20260926 07:36:40.431012 5919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:40 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408200, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:40.434206 5919 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:36:40.435770 5919 webserver.cc:460] Webserver started at http://127.22.221.65:43283/ using document root <none> and password file <none>
I20260926 07:36:40.436456 5919 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:40.436606 5919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:40.441833 5919 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:36:40.444615 5968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:40.445935 5919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:40.446106 5919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "5cde7fca027b4c41a8be28ed02bf5b65"
format_stamp: "Formatted at 2026-09-26 07:36:34 on dist-test-slave-qlq8"
server_key: "8b9bbdfad55107964a7212a375d4c24b"
server_key_iv: "d972e4f3e8d63705ff1ab1b1808234cb"
server_key_version: "encryptionkey@0"
I20260926 07:36:40.447089 5919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:40.475728 5919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:40.480046 5919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:40.480338 5919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:40.481344 5919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:40.482715 5919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:40.482870 5919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:40.482997 5919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:40.483088 5919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:40.517280 5919 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:39963
I20260926 07:36:40.517369 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:39963 every 8 connection(s)
I20260926 07:36:40.518771 5919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:36:40.523703 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:40.521889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:51015)
0926 07:36:40.522579 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:40.522815 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":273,"threads_started":1}
W20260926 07:36:40.524348 6081 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:40587 (0 consecutive failures): Network error: Failed to ping master at 127.22.221.126:40587: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
I20260926 07:36:40.525116 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:40.524818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:47027)
0926 07:36:40.524947 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:40.525013 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:36:40.525902 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:40.525611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:36041)
0926 07:36:40.525725 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:40.525799 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:36:40.526155 6081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:36:40.529031 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 5919
I20260926 07:36:40.529610 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:46843
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:40587
--webserver_port=36463
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:40.853915 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:40.854246 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:40.854422 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:40.861491 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:40.861642 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:40.861836 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:36:40.871840 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:46843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=36463
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:40.873157 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:40.874495 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:40.888568 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:40.888541 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:40.888640 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:40.889093 6084 server_base.cc:1034] running on GCE node
I20260926 07:36:40.889906 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:40.890965 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:40.892207 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790408200892124 us; error 82 us; skew 500 ppm
I20260926 07:36:40.892551 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:40 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408200, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:40.895931 6084 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:36:40.897574 6084 webserver.cc:460] Webserver started at http://127.22.221.66:36463/ using document root <none> and password file <none>
I20260926 07:36:40.898324 6084 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:40.898480 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:40.904057 6084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:36:40.906827 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:40.908252 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:36:40.908437 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "45ef4776466046418c871e49e775ca5a"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "14df5bea1539a0d5048bf6bf0f4c6232"
server_key_iv: "41add901c5e1d87f76689b0ad271c8ba"
server_key_version: "encryptionkey@0"
I20260926 07:36:40.909479 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:40.924744 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:40.928629 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:40.928920 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:40.929895 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:40.931372 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:40.931489 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:40.931644 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:40.931740 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:40.965389 6084 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:46843
I20260926 07:36:40.965472 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:46843 every 8 connection(s)
I20260926 07:36:40.966928 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:36:40.971607 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:40.969788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:56329)
0926 07:36:40.970487 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:40.970694 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":276,"threads_started":1}
I20260926 07:36:40.971817 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6084
W20260926 07:36:40.972252 6212 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:40587 (0 consecutive failures): Network error: Failed to ping master at 127.22.221.126:40587: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
I20260926 07:36:40.972478 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:37047
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:40587
--webserver_port=33601
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:36:40.973066 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:40.972739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:58979)
0926 07:36:40.972889 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:40.972967 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:36:40.973932 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:40.973601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:59867)
0926 07:36:40.973741 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:40.973821 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:36:40.974165 6212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:36:41.297567 6215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:41.297858 6215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:41.297957 6215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:41.305212 6215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:41.305320 6215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:41.305500 6215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:36:41.315882 6215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:37047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=33601
--tserver_master_addrs=127.22.221.126:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:41.317239 6215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:41.318562 6215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:41.331758 6221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:41.331831 6223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:41.331856 6220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:41.332967 6215 server_base.cc:1034] running on GCE node
I20260926 07:36:41.333554 6215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:41.334450 6215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:41.335645 6215 hybrid_clock.cc:648] HybridClock initialized: now 1790408201335576 us; error 59 us; skew 500 ppm
I20260926 07:36:41.335992 6215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:41 dist-test-slave-qlq8 krb5kdc[4767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408201, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:41.339161 6215 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:36:41.340575 6215 webserver.cc:460] Webserver started at http://127.22.221.67:33601/ using document root <none> and password file <none>
I20260926 07:36:41.341236 6215 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:41.341387 6215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:41.346809 6215 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:36:41.349665 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:41.351092 6215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:36:41.351392 6215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "9ab39e6c0db14929a891d1ac86a5d917"
format_stamp: "Formatted at 2026-09-26 07:36:35 on dist-test-slave-qlq8"
server_key: "8410112c2bb075212d02bcb4baf48dcb"
server_key_iv: "f6e1f068b5d8672e2f334735e42e3f91"
server_key_version: "encryptionkey@0"
I20260926 07:36:41.352432 6215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:41.364689 6215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:41.368616 6215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:41.368896 6215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:41.369822 6215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:41.371080 6215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:41.371198 6215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:41.371341 6215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:41.371407 6215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:41.404758 6215 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:37047
I20260926 07:36:41.404852 6342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:37047 every 8 connection(s)
I20260926 07:36:41.406350 6215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:36:41.411203 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:41.409423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:50465)
0926 07:36:41.410144 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:41.410376 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
W20260926 07:36:41.412036 6343 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:40587 (0 consecutive failures): Network error: Failed to ping master at 127.22.221.126:40587: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
I20260926 07:36:41.412855 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:41.412523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:58061)
0926 07:36:41.412663 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:41.412732 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:36:41.413699 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:41.413406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:54453)
0926 07:36:41.413521 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:41.413593 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 07:36:41.413956 6343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:36:41.416531 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6215
I20260926 07:36:41.527678 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:41.526818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:44675)
0926 07:36:41.527331 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:41.527451 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260926 07:36:41.975502 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:41.974650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:49301)
0926 07:36:41.975089 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:41.975210 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":209,"threads_started":1}
I20260926 07:36:42.415343 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:42.414510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:45629)
0926 07:36:42.414959 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:42.415063 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260926 07:36:42.529387 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:42.528685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:37929)
0926 07:36:42.529134 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:42.529228 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":218,"threads_started":1}
I20260926 07:36:42.977072 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:42.976376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:36961)
0926 07:36:42.976826 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:42.976909 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":206,"threads_started":1}
I20260926 07:36:43.417124 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:43.416372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:51723)
0926 07:36:43.416846 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:43.416941 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260926 07:36:43.531116 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:43.530376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:34593)
0926 07:36:43.530851 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:43.530938 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
I20260926 07:36:43.978673 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:43.977986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:60275)
0926 07:36:43.978432 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:43.978514 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260926 07:36:44.418737 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:44.418046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:47831)
0926 07:36:44.418484 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:44.418571 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":236,"threads_started":1}
I20260926 07:36:44.532927 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:44.532081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:42557)
0926 07:36:44.532644 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:44.532755 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260926 07:36:44.980386 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:44.979687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:39287)
0926 07:36:44.980108 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:44.980188 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":201,"threads_started":1}
I20260926 07:36:45.420408 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:45.419689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:49111)
0926 07:36:45.420109 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:45.420193 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":205,"threads_started":1}
I20260926 07:36:45.534713 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:45.533893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:58689)
0926 07:36:45.534382 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:45.534466 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
I20260926 07:36:45.981940 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:45.981248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:49717)
0926 07:36:45.981697 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:45.981777 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260926 07:36:46.421965 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:46.421304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:41395)
0926 07:36:46.421726 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:46.421804 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260926 07:36:46.536499 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:46.535811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:59323)
0926 07:36:46.536245 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:46.536342 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260926 07:36:46.983781 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:46.982915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:37221)
0926 07:36:46.983452 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:46.983582 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":248,"threads_started":1}
I20260926 07:36:47.423743 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:47.422893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:55209)
0926 07:36:47.423441 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:47.423547 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":247,"threads_started":1}
I20260926 07:36:47.538393 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:47.537506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:53475)
0926 07:36:47.538069 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:47.538188 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":252,"threads_started":1}
I20260926 07:36:47.985644 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:47.984803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:42905)
0926 07:36:47.985353 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:47.985429 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260926 07:36:48.425486 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:48.424679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:38277)
0926 07:36:48.425187 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:48.425275 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":226,"threads_started":1}
I20260926 07:36:48.540206 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:48.539356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:49747)
0926 07:36:48.539912 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:48.540021 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":282,"threads_started":1}
I20260926 07:36:48.987376 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:48.986522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:40405)
0926 07:36:48.987032 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:48.987124 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260926 07:36:49.427402 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:49.426548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:37561)
0926 07:36:49.427091 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:49.427179 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260926 07:36:49.541869 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:49.541059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:51051)
0926 07:36:49.541551 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:49.541680 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260926 07:36:49.989043 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:49.988343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:60507)
0926 07:36:49.988795 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:49.988875 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260926 07:36:50.429060 6373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:50.428359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:60053)
0926 07:36:50.428803 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:50.428883 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":209,"threads_started":1}
I20260926 07:36:50.544037 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:50.542901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:57035)
0926 07:36:50.543689 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:50.543857 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":458,"threads_started":1}
I20260926 07:36:50.990888 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:50.990041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:58467)
0926 07:36:50.990555 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:50.990646 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":290,"threads_started":1}
I20260926 07:36:51.430936 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:51.430176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:46437)
0926 07:36:51.430675 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:51.430767 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
I20260926 07:36:51.545682 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:51.544935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:39145)
0926 07:36:51.545405 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:51.545501 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260926 07:36:51.992561 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:51.991889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:51795)
0926 07:36:51.992308 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:51.992391 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
I20260926 07:36:52.433115 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:52.431969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:45219)
0926 07:36:52.432826 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:52.432922 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
I20260926 07:36:52.547365 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:52.546589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:53637)
0926 07:36:52.547063 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:52.547146 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260926 07:36:52.994130 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:52.993468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:53733)
0926 07:36:52.993891 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:52.993971 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260926 07:36:53.434675 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:53.433996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:49525)
0926 07:36:53.434407 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:53.434489 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260926 07:36:53.549001 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:53.548223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:52459)
0926 07:36:53.548720 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:53.548816 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":245,"threads_started":1}
I20260926 07:36:53.995939 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:53.995019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:34223)
0926 07:36:53.995661 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:53.995753 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":399,"threads_started":1}
I20260926 07:36:54.436471 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:54.435680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:37099)
0926 07:36:54.436186 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:54.436303 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":247,"threads_started":1}
I20260926 07:36:54.550719 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:54.549869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:40923)
0926 07:36:54.550421 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:54.550527 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":334,"threads_started":1}
I20260926 07:36:54.997629 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:54.996862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:44509)
0926 07:36:54.997360 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:54.997449 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260926 07:36:55.438127 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:55.437391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:59837)
0926 07:36:55.437863 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:55.437950 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
I20260926 07:36:55.552690 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:55.551906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.65:53777)
0926 07:36:55.552388 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:55.552505 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260926 07:36:55.999655 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:55.998796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.66:51721)
0926 07:36:55.999322 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:55.999425 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":235,"threads_started":1}
I20260926 07:36:56.422868 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 5919
I20260926 07:36:56.440694 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:56.439361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:40587 (local address 127.22.221.67:35479)
0926 07:36:56.440425 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:56.440524 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":877,"threads_started":1}
I20260926 07:36:56.446244 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:56.424191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.22.221.65:39963)
0926 07:36:56.424810 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:56.424834 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:56.424883 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:36:56.426887 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:56.426898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:56.427415 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:56.427721 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:56.429072 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:56.430320 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:56.431593 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:56.432249 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:56.432416 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:56.440351 (+ 7935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:56.440363 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:56.440382 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:56.440427 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:56.443415 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:56.444076 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:56.444082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:56.444088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:56.444246 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:56.444582 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:56.444585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:56.444587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:56.445075 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:56.445293 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:56.445667 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 07:36:56.445984 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":271,"threads_started":1}
I20260926 07:36:56.447594 6076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:56.535138 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 5919
I20260926 07:36:56.551893 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6084
I20260926 07:36:56.570212 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:56.552944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.22.221.66:46843)
0926 07:36:56.553475 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:56.553488 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:56.553523 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:36:56.555090 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:56.555099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:56.555640 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:56.556002 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:56.557528 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:56.558764 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:56.560157 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:56.560835 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:56.561035 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:56.563774 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:56.563781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:56.563795 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:56.563830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:56.566703 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:56.567495 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:56.567505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:56.567514 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:56.567746 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:56.568146 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:56.568151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:56.568155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:56.568719 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:56.569045 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:56.569494 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 07:36:56.569929 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":263,"threads_started":1}
I20260926 07:36:56.571659 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:56.655061 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6084
I20260926 07:36:56.672395 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6215
I20260926 07:36:56.690804 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:56.673499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.22.221.67:37047)
0926 07:36:56.674089 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:56.674103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:56.674147 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:36:56.675901 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:56.675911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:56.676339 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:56.676699 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:56.678130 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:56.679399 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:56.680678 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:56.681332 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:56.681521 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:56.684441 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:56.684449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:56.684464 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:56.684555 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:56.687513 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:56.688232 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:56.688241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:56.688255 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:56.688414 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:56.688816 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:56.688823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:56.688826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:56.689333 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:56.689655 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:56.690097 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 07:36:56.690509 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":291,"threads_started":1}
I20260926 07:36:56.692322 6338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:36:56.779162 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6215
2026-09-26T07:36:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24439 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:36:56 dist-test-slave-qlq8 krb5kdc[6400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:36:56 dist-test-slave-qlq8 krb5kdc[6400](info): set up 2 sockets
Sep 26 07:36:56 dist-test-slave-qlq8 krb5kdc[6400](info): commencing operation
krb5kdc: starting...
W20260926 07:36:57.957959 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:36:58 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:36:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:36:58.207154 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37657
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:58.532757 6413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:58.533229 6413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:58.533386 6413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:58.540913 6413 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:36:58.541023 6413 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:36:58.541092 6413 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:36:58.541169 6413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:36:58.541258 6413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:58.541370 6413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:58.541461 6413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:36:58.541543 6413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:36:58.551780 6413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:58.553139 6413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:58.554579 6413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:58.566751 6418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:58.566856 6419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:58.566923 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:58.568040 6413 server_base.cc:1034] running on GCE node
I20260926 07:36:58.568864 6413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:58.570279 6413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:58.571542 6413 hybrid_clock.cc:648] HybridClock initialized: now 1790408218571436 us; error 36 us; skew 500 ppm
I20260926 07:36:58.571899 6413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:58 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408218, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:58.575085 6413 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:36:58.576752 6413 webserver.cc:460] Webserver started at http://127.22.221.126:36895/ using document root <none> and password file <none>
I20260926 07:36:58.577481 6413 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:58.577616 6413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:58.577939 6413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:58.581053 6413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "a1f30b2127cb45f8bb90ba9a3888ea16"
format_stamp: "Formatted at 2026-09-26 07:36:58 on dist-test-slave-qlq8"
server_key: "80d2c334165b534d68b9b3cd7225a143"
server_key_iv: "8b59e91ac2b8b89a2420f880fa52cf5b"
server_key_version: "encryptionkey@0"
I20260926 07:36:58.582134 6413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "a1f30b2127cb45f8bb90ba9a3888ea16"
format_stamp: "Formatted at 2026-09-26 07:36:58 on dist-test-slave-qlq8"
server_key: "80d2c334165b534d68b9b3cd7225a143"
server_key_iv: "8b59e91ac2b8b89a2420f880fa52cf5b"
server_key_version: "encryptionkey@0"
I20260926 07:36:58.587471 6413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:36:58.590880 6428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:58.592329 6413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:58.592509 6413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "a1f30b2127cb45f8bb90ba9a3888ea16"
format_stamp: "Formatted at 2026-09-26 07:36:58 on dist-test-slave-qlq8"
server_key: "80d2c334165b534d68b9b3cd7225a143"
server_key_iv: "8b59e91ac2b8b89a2420f880fa52cf5b"
server_key_version: "encryptionkey@0"
I20260926 07:36:58.592741 6413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:58.617367 6413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:58.621649 6413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:58.621924 6413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:58.639487 6413 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37657
I20260926 07:36:58.639585 6479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37657 every 8 connection(s)
I20260926 07:36:58.640878 6413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:36:58.644722 6480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:36:58.650014 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6413
I20260926 07:36:58.650233 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:36:58.650992 23413 external_mini_cluster.cc:1428] Setting key aaf8e91e3c717967429399e7580f8b69
I20260926 07:36:58.656191 6480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap starting.
I20260926 07:36:58.659832 6480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Neither blocks nor log segments found. Creating new log.
I20260926 07:36:58.660801 6480 log.cc:824] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Log is configured to *not* fsync() on all Append() calls
I20260926 07:36:58.663625 6480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: No bootstrap required, opened a new log
Sep 26 07:36:58 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:58.671769 6480 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:36:58.672147 6480 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:36:58.672268 6480 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f30b2127cb45f8bb90ba9a3888ea16, State: Initialized, Role: FOLLOWER
I20260926 07:36:58.673098 6480 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:36:58.673367 6480 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:36:58.673517 6480 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:36:58.673717 6480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:36:58.674533 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:58.653214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.22.221.126:37657)
0926 07:36:58.654076 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:58.654088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:58.654137 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:36:58.655883 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:58.655905 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:58.656323 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:58.656652 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:58.658481 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:58.659850 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:58.661244 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:58.661936 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:58.662136 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:58.666437 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:58.666445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:58.666462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:58.666498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:58.669796 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:58.670582 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:58.670591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:58.670597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:58.670771 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:58.671174 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:58.671180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:58.671183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:58.671789 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:58.672105 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:58.672670 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 07:36:58.673123 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":416,"threads_started":1}
I20260926 07:36:58.676342 6480 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:36:58.676839 6480 leader_election.cc:302] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f30b2127cb45f8bb90ba9a3888ea16; no voters:
I20260926 07:36:58.677474 6480 leader_election.cc:288] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:36:58.677577 6485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:36:58.678257 6485 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 LEADER]: Becoming Leader. State: Replica: a1f30b2127cb45f8bb90ba9a3888ea16, State: Running, Role: LEADER
I20260926 07:36:58.678791 6485 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:36:58.679770 6480 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:36:58.681638 6486 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } } }
I20260926 07:36:58.681732 6487 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f30b2127cb45f8bb90ba9a3888ea16. Latest consensus state: current_term: 1 leader_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } } }
I20260926 07:36:58.682107 6486 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:58.682137 6487 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:36:58.684508 6490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:36:58.689013 6490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:36:58.697355 6490 catalog_manager.cc:1347] Generated new cluster ID: 25c96893f57b45a8aa9ff78f62fc17c8
I20260926 07:36:58.697487 6490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:36:58.737294 6490 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:36:58.738706 6490 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:36:58.750756 6490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Generated new TSK 0
I20260926 07:36:58.751727 6490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:36:58.818910 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37657
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:59.154554 6506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:59.154824 6506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:59.154982 6506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:59.162425 6506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:59.162523 6506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:59.162751 6506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:36:59.172649 6506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:59.174211 6506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:59.175632 6506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:59.188657 6514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:59.188791 6511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:59.188791 6512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:59.189893 6506 server_base.cc:1034] running on GCE node
I20260926 07:36:59.190418 6506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:59.191337 6506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:59.192528 6506 hybrid_clock.cc:648] HybridClock initialized: now 1790408219192517 us; error 62 us; skew 500 ppm
I20260926 07:36:59.192858 6506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:59 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408219, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:59.195926 6506 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:36:59.197467 6506 webserver.cc:460] Webserver started at http://127.22.221.65:44807/ using document root <none> and password file <none>
I20260926 07:36:59.198165 6506 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:59.198309 6506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:59.198617 6506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:59.201464 6506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "d976edb7260c4ddca6e6ee4c5475ccd3"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "5e5064763574a90c4d625bd5e44c850b"
server_key_iv: "2235b86dc444542aed5d178dec471d38"
server_key_version: "encryptionkey@0"
I20260926 07:36:59.202214 6506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "d976edb7260c4ddca6e6ee4c5475ccd3"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "5e5064763574a90c4d625bd5e44c850b"
server_key_iv: "2235b86dc444542aed5d178dec471d38"
server_key_version: "encryptionkey@0"
I20260926 07:36:59.207041 6506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 07:36:59.210414 6521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:59.211905 6506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:36:59.212088 6506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "d976edb7260c4ddca6e6ee4c5475ccd3"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "5e5064763574a90c4d625bd5e44c850b"
server_key_iv: "2235b86dc444542aed5d178dec471d38"
server_key_version: "encryptionkey@0"
I20260926 07:36:59.212297 6506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:59.228008 6506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:59.232111 6506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:59.232402 6506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:59.233379 6506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:59.234722 6506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:59.234826 6506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:59.234980 6506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:59.235054 6506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:59.268827 6506 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:38743
I20260926 07:36:59.268873 6633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:38743 every 8 connection(s)
I20260926 07:36:59.270104 6506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:36:59.276463 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6506
I20260926 07:36:59.276685 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:36:59.277347 23413 external_mini_cluster.cc:1428] Setting key 747a4e5c1f5e8326674871ffce66af21
Sep 26 07:36:59 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408219, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:59.292397 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:59.272747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:44671 (local address 127.22.221.126:37657)
0926 07:36:59.273159 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:59.273163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:59.274228 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 07:36:59.276127 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:59.276133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:59.276222 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:59.276342 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:59.277975 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.278893 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.280762 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.281517 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.281651 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:59.285622 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:59.285627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:59.285630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:59.285663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:59.289043 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:59.289767 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:59.289776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:59.289782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:59.289935 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:59.290380 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:59.290387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:59.290393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:59.290780 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:59.290948 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:59.291665 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 07:36:59.291930 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":208,"threads_started":1}
I20260926 07:36:59.293004 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:59.273238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:44671)
0926 07:36:59.273975 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:59.274015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:59.275862 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:59.276632 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:59.276640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:59.277185 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:59.277786 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:59.277803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.279124 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.279129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:59.280627 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:59.280639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.281727 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.281732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:59.281916 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:59.282600 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:59.282622 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:59.285349 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:59.289198 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:59.289205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:59.289217 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:59.289628 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:59.290054 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:59.290057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:59.290060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:59.290256 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:59.290901 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:59.290906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:59.291259 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 07:36:59.291499 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 07:36:59.291879 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":307,"threads_started":1}
I20260926 07:36:59.294555 6634 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37657
I20260926 07:36:59.294889 6634 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:59.295723 6634 heartbeater.cc:511] Master 127.22.221.126:37657 requested a full tablet report, sending...
I20260926 07:36:59.298062 6445 ts_manager.cc:194] Registered new tserver with Master: d976edb7260c4ddca6e6ee4c5475ccd3 (127.22.221.65:38743)
I20260926 07:36:59.299832 6445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:44671
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:36:59.333079 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37657
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:36:59.656373 6640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:36:59.656682 6640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:36:59.656854 6640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:36:59.664090 6640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:36:59.664229 6640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:36:59.664402 6640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:36:59.674045 6640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:36:59.675491 6640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:36:59.676780 6640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:36:59.689949 6645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:59.689973 6646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:36:59.690337 6648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:36:59.691109 6640 server_base.cc:1034] running on GCE node
I20260926 07:36:59.691824 6640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:36:59.692719 6640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:36:59.693943 6640 hybrid_clock.cc:648] HybridClock initialized: now 1790408219693894 us; error 31 us; skew 500 ppm
I20260926 07:36:59.694285 6640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:36:59 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408219, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:36:59.697616 6640 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:36:59.699064 6640 webserver.cc:460] Webserver started at http://127.22.221.66:41809/ using document root <none> and password file <none>
I20260926 07:36:59.699781 6640 fs_manager.cc:359] Metadata directory not provided
I20260926 07:36:59.699921 6640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:36:59.700223 6640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:36:59.703054 6640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "fd2b56eeaf3f428088e7f32695f6d1c2"
server_key_iv: "77cc01eeffb7589bd3a179cf0aedfb86"
server_key_version: "encryptionkey@0"
I20260926 07:36:59.703877 6640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "fd2b56eeaf3f428088e7f32695f6d1c2"
server_key_iv: "77cc01eeffb7589bd3a179cf0aedfb86"
server_key_version: "encryptionkey@0"
I20260926 07:36:59.708896 6640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:36:59.712392 6655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:59.713634 6640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:36:59.713801 6640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "fd2b56eeaf3f428088e7f32695f6d1c2"
server_key_iv: "77cc01eeffb7589bd3a179cf0aedfb86"
server_key_version: "encryptionkey@0"
I20260926 07:36:59.713997 6640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:36:59.742158 6640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:36:59.746256 6640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:36:59.746582 6640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:36:59.747579 6640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:36:59.748904 6640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:36:59.749007 6640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:59.749150 6640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:36:59.749220 6640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:36:59.782761 6640 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:41823
I20260926 07:36:59.782852 6767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:41823 every 8 connection(s)
I20260926 07:36:59.784161 6640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:36:59.785844 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6640
I20260926 07:36:59.786032 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:36:59.786638 23413 external_mini_cluster.cc:1428] Setting key d7017cc4851568aaa2cdd90cbfdcfbe8
Sep 26 07:36:59 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408219, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:36:59.806746 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:59.787121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:55603 (local address 127.22.221.126:37657)
0926 07:36:59.787366 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 07:36:59.787371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:36:59.788775 (+ 1404us) server_negotiation.cc:407] Connection header received
0926 07:36:59.790734 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:36:59.790740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:36:59.790836 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:36:59.790959 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:36:59.792580 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.793285 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.794976 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.795756 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.795899 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:59.799654 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:36:59.799662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:36:59.799670 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:36:59.799716 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:36:59.803447 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:59.804158 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:59.804166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:59.804172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:59.804327 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:36:59.804752 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:36:59.804760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:36:59.804766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:36:59.805153 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:36:59.805302 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:36:59.805975 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 07:36:59.806262 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:36:59.807480 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:36:59.787820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:55603)
0926 07:36:59.788628 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 07:36:59.788674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:36:59.790451 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:36:59.791208 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:36:59.791217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:36:59.791829 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:36:59.792389 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:59.792406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.793440 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.793446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:59.794813 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:36:59.794829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:36:59.795974 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:36:59.795979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:36:59.796197 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:36:59.796968 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:36:59.796991 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:36:59.799404 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:36:59.803580 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:59.803587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:59.803602 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:59.804016 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:59.804432 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:36:59.804435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:36:59.804438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:36:59.804622 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:36:59.805271 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:36:59.805276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:36:59.805590 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 07:36:59.805824 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 07:36:59.806209 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":352,"threads_started":1}
I20260926 07:36:59.809041 6768 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37657
I20260926 07:36:59.809371 6768 heartbeater.cc:464] Registering TS with master...
I20260926 07:36:59.810078 6768 heartbeater.cc:511] Master 127.22.221.126:37657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:36:59.811605 6445 ts_manager.cc:194] Registered new tserver with Master: d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823)
I20260926 07:36:59.812457 6445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:55603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:36:59.842723 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37657
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:37:00.170034 6773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:00.170300 6773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:00.170462 6773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:00.177896 6773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:00.177999 6773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:00.178189 6773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:37:00.188035 6773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:00.189241 6773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:00.190505 6773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:00.203115 6779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:00.203444 6781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:00.203145 6778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:00.204210 6773 server_base.cc:1034] running on GCE node
I20260926 07:37:00.204798 6773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:00.205674 6773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:00.206894 6773 hybrid_clock.cc:648] HybridClock initialized: now 1790408220206846 us; error 57 us; skew 500 ppm
I20260926 07:37:00.207207 6773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:00 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408220, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:00.210331 6773 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:37:00.211735 6773 webserver.cc:460] Webserver started at http://127.22.221.67:38673/ using document root <none> and password file <none>
I20260926 07:37:00.212410 6773 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:00.212507 6773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:00.212764 6773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:37:00.215583 6773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "0a6f96ca0437455da919c231f906527b"
format_stamp: "Formatted at 2026-09-26 07:37:00 on dist-test-slave-qlq8"
server_key: "a397a57b0c7a6908e8ba95dd897012ac"
server_key_iv: "934935dd07a77f6d6c7e82e75d27b2f0"
server_key_version: "encryptionkey@0"
I20260926 07:37:00.216303 6773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "0a6f96ca0437455da919c231f906527b"
format_stamp: "Formatted at 2026-09-26 07:37:00 on dist-test-slave-qlq8"
server_key: "a397a57b0c7a6908e8ba95dd897012ac"
server_key_iv: "934935dd07a77f6d6c7e82e75d27b2f0"
server_key_version: "encryptionkey@0"
I20260926 07:37:00.221336 6773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:37:00.224846 6788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:00.226213 6773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:37:00.226414 6773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "0a6f96ca0437455da919c231f906527b"
format_stamp: "Formatted at 2026-09-26 07:37:00 on dist-test-slave-qlq8"
server_key: "a397a57b0c7a6908e8ba95dd897012ac"
server_key_iv: "934935dd07a77f6d6c7e82e75d27b2f0"
server_key_version: "encryptionkey@0"
I20260926 07:37:00.226634 6773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:00.266769 6773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:00.271768 6773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:00.272192 6773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:00.273200 6773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:00.274565 6773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:00.274688 6773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:00.274853 6773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:00.274945 6773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:00.302865 6634 heartbeater.cc:503] Master 127.22.221.126:37657 was elected leader, sending a full tablet report...
I20260926 07:37:00.310889 6773 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:44717
I20260926 07:37:00.310940 6900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:44717 every 8 connection(s)
I20260926 07:37:00.312314 6773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:37:00.316196 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6773
I20260926 07:37:00.316399 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:37:00.317023 23413 external_mini_cluster.cc:1428] Setting key 89bd8f5126504322c290bff7a35a3886
Sep 26 07:37:00 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408220, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:37:00.333192 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.314764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:56269 (local address 127.22.221.126:37657)
0926 07:37:00.315354 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.315359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.316383 (+ 1024us) server_negotiation.cc:407] Connection header received
0926 07:37:00.318899 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.318903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.318967 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.319098 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:00.320918 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.321695 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.323308 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.323718 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.323822 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.327189 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:00.327194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:00.327197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:00.327281 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:00.329734 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:00.330479 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:00.330483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:00.330486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:00.330649 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:00.331166 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:00.331170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:00.331172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:00.331682 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:00.331851 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.332682 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.332919 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":309,"threads_started":1}
I20260926 07:37:00.334136 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.315185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:56269)
0926 07:37:00.316067 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:00.316132 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:00.318588 (+ 2456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:00.319371 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:00.319383 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:00.319974 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:37:00.320758 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.320778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.321853 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.321858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.323126 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.323138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.323921 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.323926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.324152 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.324867 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:37:00.324896 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:37:00.326984 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:37:00.329859 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:00.329865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:00.329876 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:00.330299 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:00.330776 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:00.330780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:00.330784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:00.330988 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:00.331825 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:37:00.331833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:37:00.332244 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 07:37:00.332486 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 07:37:00.333056 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":281,"threads_started":1}
I20260926 07:37:00.335484 6901 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37657
I20260926 07:37:00.335762 6901 heartbeater.cc:464] Registering TS with master...
I20260926 07:37:00.336455 6901 heartbeater.cc:511] Master 127.22.221.126:37657 requested a full tablet report, sending...
I20260926 07:37:00.337814 6445 ts_manager.cc:194] Registered new tserver with Master: 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717)
I20260926 07:37:00.338631 6445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:56269
I20260926 07:37:00.345047 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:37:00.361865 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.349519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.22.221.126:37657)
0926 07:37:00.349721 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.349726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.350256 (+ 530us) server_negotiation.cc:407] Connection header received
0926 07:37:00.350393 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.350397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.350470 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.350553 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:00.351987 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.352727 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.354165 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.354694 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.354808 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.357206 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:00.357211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:00.357214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:00.357243 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:00.359695 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:00.360315 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:00.360319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:00.360321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:00.360430 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:00.360762 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:00.360765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:00.360767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:00.361051 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:00.361226 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.361489 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.361651 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:37:00.365829 6445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:37:00.372649 6445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:37:00.397691 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.384953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.22.221.66:41823)
0926 07:37:00.385429 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.385438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.385467 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:37:00.385797 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.385805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.386088 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.386350 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:00.387464 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.389500 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.393097 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.395091 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.395260 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.395478 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.397230 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.397422 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":918,"threads_started":1}
I20260926 07:37:00.397859 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.384477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41823 (local address 127.0.0.1:45242)
0926 07:37:00.385338 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:00.385377 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:00.385579 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:00.386665 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:00.386676 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:00.386711 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:00.387170 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.387188 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.389725 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.389734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.392862 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.392883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.395618 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.395623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.395939 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.396011 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 07:37:00.396169 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 07:37:00.396420 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":617,"threads_started":1}
I20260926 07:37:00.401760 6703 tablet_service.cc:1467] Processing CreateTablet for tablet d67d5c5ac326473381e73ce5526d36f2 (DEFAULT_TABLE table=test-table [id=53de9120ab6a4a2ba9f89af0d38d8e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:37:00.403285 6703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67d5c5ac326473381e73ce5526d36f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:37:00.405624 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.385866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:44717 (local address 127.0.0.1:58228)
0926 07:37:00.386508 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:00.386530 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:00.386671 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:00.398672 (+ 12001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:00.398676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:00.398702 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:00.399195 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.399209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.401288 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.401296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.403005 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.403017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.404750 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.404756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.404946 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.404979 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:37:00.405081 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:37:00.405202 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":113,"thread_start_us":332,"threads_started":1}
I20260926 07:37:00.405687 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.386274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.22.221.67:44717)
0926 07:37:00.398129 (+ 11855us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.398137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.398168 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:37:00.398267 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.398275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.398497 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.398700 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:00.399442 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.401115 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.403160 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.404546 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.404677 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.404888 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.405145 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.405315 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11676,"thread_start_us":11492,"threads_started":1}
I20260926 07:37:00.406706 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.386501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:38743 (local address 127.0.0.1:53566)
0926 07:37:00.397268 (+ 10767us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:00.397292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:00.397438 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:00.398798 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:00.398804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:00.398830 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:00.399437 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.399450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.401324 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.401332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.403602 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.403615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.405680 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.405687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.406041 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.406079 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:37:00.406164 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:37:00.406292 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10636,"mutex_wait_us":97,"thread_start_us":10449,"threads_started":1}
I20260926 07:37:00.407835 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.387048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.22.221.65:38743)
0926 07:37:00.395612 (+ 8564us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.395621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.398210 (+ 2589us) server_negotiation.cc:407] Connection header received
0926 07:37:00.398319 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.398326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.398567 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.399131 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:00.399657 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.401187 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.403759 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.405178 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.405305 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.405518 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.407330 (+ 1812us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.407499 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8366,"thread_start_us":8218,"threads_started":1}
I20260926 07:37:00.409655 6836 tablet_service.cc:1467] Processing CreateTablet for tablet d67d5c5ac326473381e73ce5526d36f2 (DEFAULT_TABLE table=test-table [id=53de9120ab6a4a2ba9f89af0d38d8e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:37:00.411185 6836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67d5c5ac326473381e73ce5526d36f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:37:00.411392 6569 tablet_service.cc:1467] Processing CreateTablet for tablet d67d5c5ac326473381e73ce5526d36f2 (DEFAULT_TABLE table=test-table [id=53de9120ab6a4a2ba9f89af0d38d8e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:37:00.412690 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67d5c5ac326473381e73ce5526d36f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:37:00.415122 6920 tablet_bootstrap.cc:492] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Bootstrap starting.
I20260926 07:37:00.418515 6920 tablet_bootstrap.cc:654] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Neither blocks nor log segments found. Creating new log.
I20260926 07:37:00.419544 6920 log.cc:824] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:00.422099 6920 tablet_bootstrap.cc:492] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: No bootstrap required, opened a new log
I20260926 07:37:00.422394 6920 ts_tablet_manager.cc:1397] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260926 07:37:00.425523 6922 tablet_bootstrap.cc:492] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Bootstrap starting.
I20260926 07:37:00.427032 6923 tablet_bootstrap.cc:492] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Bootstrap starting.
I20260926 07:37:00.429394 6922 tablet_bootstrap.cc:654] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Neither blocks nor log segments found. Creating new log.
I20260926 07:37:00.429773 6923 tablet_bootstrap.cc:654] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Neither blocks nor log segments found. Creating new log.
I20260926 07:37:00.430652 6922 log.cc:824] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:00.430923 6923 log.cc:824] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:00.432132 6920 raft_consensus.cc:348] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.432667 6920 raft_consensus.cc:374] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:37:00.432790 6920 raft_consensus.cc:729] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b33f5f3f6c4abe9c991df6f68cedfd, State: Initialized, Role: FOLLOWER
I20260926 07:37:00.433506 6920 consensus_queue.cc:260] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.433908 6922 tablet_bootstrap.cc:492] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: No bootstrap required, opened a new log
I20260926 07:37:00.433908 6923 tablet_bootstrap.cc:492] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: No bootstrap required, opened a new log
I20260926 07:37:00.434185 6922 ts_tablet_manager.cc:1397] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 07:37:00.434178 6923 ts_tablet_manager.cc:1397] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:37:00.434831 6768 heartbeater.cc:503] Master 127.22.221.126:37657 was elected leader, sending a full tablet report...
I20260926 07:37:00.435566 6920 ts_tablet_manager.cc:1428] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260926 07:37:00.445132 6922 raft_consensus.cc:348] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.445124 6923 raft_consensus.cc:348] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.445628 6923 raft_consensus.cc:374] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:37:00.445617 6922 raft_consensus.cc:374] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:37:00.445731 6922 raft_consensus.cc:729] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6f96ca0437455da919c231f906527b, State: Initialized, Role: FOLLOWER
I20260926 07:37:00.445731 6923 raft_consensus.cc:729] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d976edb7260c4ddca6e6ee4c5475ccd3, State: Initialized, Role: FOLLOWER
I20260926 07:37:00.446404 6922 consensus_queue.cc:260] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.446409 6923 consensus_queue.cc:260] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.448138 6901 heartbeater.cc:503] Master 127.22.221.126:37657 was elected leader, sending a full tablet report...
I20260926 07:37:00.448395 6923 ts_tablet_manager.cc:1428] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 07:37:00.449623 6922 ts_tablet_manager.cc:1428] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260926 07:37:00.522470 6635 tablet.cc:2369] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:37:00.536352 6769 tablet.cc:2369] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:37:00.564173 6902 tablet.cc:2369] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:37:00.702595 6928 raft_consensus.cc:482] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:37:00.702950 6928 raft_consensus.cc:504] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.704833 6928 leader_election.cc:288] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823), 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717)
I20260926 07:37:00.714164 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.705196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:46205 (local address 127.22.221.66:41823)
0926 07:37:00.705405 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.705412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.705808 (+ 396us) server_negotiation.cc:407] Connection header received
0926 07:37:00.706174 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.706185 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.706289 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.706367 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:00.707342 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.708638 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.711654 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.712882 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.713018 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.713086 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.713754 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.713890 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:37:00.714164 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.705997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:53465 (local address 127.22.221.67:44717)
0926 07:37:00.706215 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.706222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.706789 (+ 567us) server_negotiation.cc:407] Connection header received
0926 07:37:00.707006 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.707011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.707077 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.707171 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:00.707809 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.708581 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.711551 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.712822 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.713003 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.713087 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.713754 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.713891 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:37:00.714490 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.705825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:44717 (local address 127.22.221.65:53465)
0926 07:37:00.706669 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:00.706683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:00.706864 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:00.707185 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:00.707189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:00.707211 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:00.707607 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.707619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.708728 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.708736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.711348 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.711363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.713127 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.713134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.713444 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.713604 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 07:37:00.713703 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:37:00.713947 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":333,"thread_start_us":534,"threads_started":1}
I20260926 07:37:00.714516 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.705030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41823 (local address 127.22.221.65:46205)
0926 07:37:00.705701 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:00.705722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:00.705938 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:00.706599 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:00.706607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:00.706641 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:00.707110 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.707123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.708778 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.708789 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.711347 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:00.711363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.713134 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.713140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:00.713446 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.713610 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 07:37:00.713703 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:37:00.713947 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":253,"spinlock_wait_cycles":44800,"thread_start_us":349,"threads_started":1}
I20260926 07:37:00.715583 6856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d5c5ac326473381e73ce5526d36f2" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6f96ca0437455da919c231f906527b" is_pre_election: true
I20260926 07:37:00.715588 6723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d5c5ac326473381e73ce5526d36f2" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" is_pre_election: true
I20260926 07:37:00.716130 6723 raft_consensus.cc:2454] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 0.
I20260926 07:37:00.716135 6856 raft_consensus.cc:2454] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 0.
I20260926 07:37:00.716756 6525 leader_election.cc:302] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a6f96ca0437455da919c231f906527b, d976edb7260c4ddca6e6ee4c5475ccd3; no voters:
I20260926 07:37:00.717206 6928 raft_consensus.cc:2781] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:37:00.717357 6928 raft_consensus.cc:482] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:37:00.717483 6928 raft_consensus.cc:3037] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:37:00.719749 6928 raft_consensus.cc:504] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.720358 6928 leader_election.cc:288] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 election: Requested vote from peers d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823), 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717)
I20260926 07:37:00.720831 6723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d5c5ac326473381e73ce5526d36f2" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
I20260926 07:37:00.720887 6856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67d5c5ac326473381e73ce5526d36f2" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6f96ca0437455da919c231f906527b"
I20260926 07:37:00.721021 6723 raft_consensus.cc:3037] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:37:00.721131 6856 raft_consensus.cc:3037] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:37:00.723206 6723 raft_consensus.cc:2454] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 1.
I20260926 07:37:00.723220 6856 raft_consensus.cc:2454] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 1.
I20260926 07:37:00.723668 6525 leader_election.cc:302] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2b33f5f3f6c4abe9c991df6f68cedfd, d976edb7260c4ddca6e6ee4c5475ccd3; no voters:
I20260926 07:37:00.723953 6928 raft_consensus.cc:2781] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:37:00.724414 6928 raft_consensus.cc:686] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 LEADER]: Becoming Leader. State: Replica: d976edb7260c4ddca6e6ee4c5475ccd3, State: Running, Role: LEADER
I20260926 07:37:00.724972 6928 consensus_queue.cc:237] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:00.729560 6445 catalog_manager.cc:5495] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d976edb7260c4ddca6e6ee4c5475ccd3 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } health_report { overall_health: HEALTHY } } }
I20260926 07:37:00.762106 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.755827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.22.221.65:38743)
0926 07:37:00.756092 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.756099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.756121 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:37:00.756365 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.756370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.756474 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.756658 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:37:00.757068 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.757894 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.759688 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.760156 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.760310 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.760862 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:37:00.761433 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:37:00.761623 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.761690 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.761783 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 07:37:00.771014 6856 raft_consensus.cc:1261] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Refusing update from remote peer d976edb7260c4ddca6e6ee4c5475ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:37:00.771034 6723 raft_consensus.cc:1261] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Refusing update from remote peer d976edb7260c4ddca6e6ee4c5475ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:37:00.772094 6928 consensus_queue.cc:1035] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:37:00.772400 6931 consensus_queue.cc:1035] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:37:00.785076 6934 mvcc.cc:205] Tried to move back new op lower bound from 7333512072266428416 to 7333512072093261824. Current Snapshot: MvccSnapshot[applied={T|T < 7333512072266428416}]
I20260926 07:37:00.813382 6445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:37:00.813640 6445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:37:00.817412 6445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Sending DeleteTablet for 3 replicas of tablet d67d5c5ac326473381e73ce5526d36f2
I20260926 07:37:00.818516 6703 tablet_service.cc:1514] Processing DeleteTablet for tablet d67d5c5ac326473381e73ce5526d36f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:37:00 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:45242
I20260926 07:37:00.818724 6836 tablet_service.cc:1514] Processing DeleteTablet for tablet d67d5c5ac326473381e73ce5526d36f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:37:00 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:58228
I20260926 07:37:00.818917 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6506
I20260926 07:37:00.819727 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet d67d5c5ac326473381e73ce5526d36f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:37:00 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:53566
I20260926 07:37:00.842767 6946 tablet_replica.cc:330] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: stopping tablet replica
I20260926 07:37:00.843888 6946 raft_consensus.cc:2229] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:37:00.844673 6944 tablet_replica.cc:330] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: stopping tablet replica
I20260926 07:37:00.845594 6946 raft_consensus.cc:2258] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:37:00.846325 6944 raft_consensus.cc:2229] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:37:00.847004 6944 raft_consensus.cc:2258] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:37:00.850107 6946 ts_tablet_manager.cc:1905] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:37:00.851296 6943 tablet_replica.cc:330] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: stopping tablet replica
I20260926 07:37:00.852526 6944 ts_tablet_manager.cc:1905] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:37:00.852667 6943 raft_consensus.cc:2229] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:37:00.853276 6943 raft_consensus.cc:2258] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:37:00.860102 6944 ts_tablet_manager.cc:1918] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:37:00.860368 6944 log.cc:1196] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/d67d5c5ac326473381e73ce5526d36f2
I20260926 07:37:00.860981 6944 ts_tablet_manager.cc:1939] T d67d5c5ac326473381e73ce5526d36f2 P 0a6f96ca0437455da919c231f906527b: Deleting consensus metadata
Sep 26 07:37:00 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:37:00.863142 6432 catalog_manager.cc:4841] TS 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717): tablet d67d5c5ac326473381e73ce5526d36f2 (table test-table [id=53de9120ab6a4a2ba9f89af0d38d8e99]) successfully deleted
I20260926 07:37:00.864230 6946 ts_tablet_manager.cc:1918] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:37:00.864401 6943 ts_tablet_manager.cc:1905] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:37:00.864516 6946 log.cc:1196] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/d67d5c5ac326473381e73ce5526d36f2
I20260926 07:37:00.865492 6946 ts_tablet_manager.cc:1939] T d67d5c5ac326473381e73ce5526d36f2 P d976edb7260c4ddca6e6ee4c5475ccd3: Deleting consensus metadata
I20260926 07:37:00.868132 6432 catalog_manager.cc:4841] TS d976edb7260c4ddca6e6ee4c5475ccd3 (127.22.221.65:38743): tablet d67d5c5ac326473381e73ce5526d36f2 (table test-table [id=53de9120ab6a4a2ba9f89af0d38d8e99]) successfully deleted
I20260926 07:37:00.871388 6943 ts_tablet_manager.cc:1918] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:37:00.871673 6943 log.cc:1196] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/d67d5c5ac326473381e73ce5526d36f2
I20260926 07:37:00.872195 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.820007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.22.221.65:38743)
0926 07:37:00.820428 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.820436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.852060 (+ 31624us) server_negotiation.cc:407] Connection header received
0926 07:37:00.852287 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.852292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.852383 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.852480 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:00.854590 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.855523 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.857487 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.858213 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.858314 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:00.864229 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:00.864248 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:00.864267 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:00.864321 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:00.868593 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:00.869596 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:00.869605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:00.869609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:00.869788 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:00.870259 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:00.870265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:00.870268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:00.870716 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:00.871007 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:00.871464 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 07:37:00.871746 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260926 07:37:00.872255 6943 ts_tablet_manager.cc:1939] T d67d5c5ac326473381e73ce5526d36f2 P d2b33f5f3f6c4abe9c991df6f68cedfd: Deleting consensus metadata
I20260926 07:37:00.872884 6629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:00.878631 6432 catalog_manager.cc:4841] TS d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823): tablet d67d5c5ac326473381e73ce5526d36f2 (table test-table [id=53de9120ab6a4a2ba9f89af0d38d8e99]) successfully deleted
I20260926 07:37:00.973469 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6506
I20260926 07:37:00.992329 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6640
Sep 26 07:37:00 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:37:01.007382 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:00.993079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.22.221.66:41823)
0926 07:37:00.993251 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:00.993255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:00.993291 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:37:00.993455 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:00.993459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:00.993520 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:00.993646 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:00.994960 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.995723 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.997230 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:00.997755 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:00.997864 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:01.002230 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:01.002243 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:01.002257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:01.002294 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:01.005007 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:01.005632 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:01.005639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:01.005641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:01.005767 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:01.006085 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:01.006088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:01.006090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:01.006419 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:01.006675 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:01.006902 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:37:01.007053 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:37:01.008080 6763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:01.101369 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6640
I20260926 07:37:01.118430 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6773
Sep 26 07:37:01 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:37:01.134348 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:01.119266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.22.221.67:44717)
0926 07:37:01.119450 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:01.119456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:01.119537 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:37:01.119751 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:01.119763 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:01.119829 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:01.119944 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:01.121312 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:01.122035 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:01.123546 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:01.123933 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:01.124069 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:01.128869 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:01.128883 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:01.128899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:01.128936 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:01.131952 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:01.132590 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:01.132597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:01.132599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:01.132728 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:01.133063 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:01.133066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:01.133068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:01.133414 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:01.133662 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:01.133934 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:37:01.134090 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:37:01.134975 6896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:01.223644 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6773
I20260926 07:37:01.240170 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6413
I20260926 07:37:01.241293 6475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:01.351567 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6413
I20260926 07:37:21.373545 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37657
--webserver_interface=127.22.221.126
--webserver_port=36895
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:37:21.707573 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:21.707927 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:21.708087 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:21.715440 6951 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:37:21.715580 6951 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:37:21.715652 6951 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:37:21.715728 6951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:37:21.715831 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:21.715932 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:21.716022 6951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:37:21.716102 6951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:37:21.726133 6951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=36895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:21.727602 6951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:21.729158 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:21.752604 6960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:21.752660 6957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:21.752909 6951 server_base.cc:1034] running on GCE node
W20260926 07:37:21.752624 6958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:21.753988 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:21.755764 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:21.757045 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790408241756977 us; error 41 us; skew 500 ppm
I20260926 07:37:21.757457 6951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:21 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408241, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:21.761276 6951 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:37:21.762969 6951 webserver.cc:460] Webserver started at http://127.22.221.126:36895/ using document root <none> and password file <none>
I20260926 07:37:21.763850 6951 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:21.764006 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:21.770223 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:37:21.773593 6967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:21.775146 6951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:37:21.775426 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "a1f30b2127cb45f8bb90ba9a3888ea16"
format_stamp: "Formatted at 2026-09-26 07:36:58 on dist-test-slave-qlq8"
server_key: "80d2c334165b534d68b9b3cd7225a143"
server_key_iv: "8b59e91ac2b8b89a2420f880fa52cf5b"
server_key_version: "encryptionkey@0"
I20260926 07:37:21.776646 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:21.807375 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:21.812543 6951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:21.812870 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:21.834020 6951 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37657
I20260926 07:37:21.834029 7018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37657 every 8 connection(s)
I20260926 07:37:21.835901 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:37:21.841235 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 6951
I20260926 07:37:21.842235 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:38743
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:37657
--webserver_port=44807
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:37:21.844003 7019 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:37:21.849151 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap starting.
I20260926 07:37:21.869288 7019 log.cc:824] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:21.881135 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:37:21.881779 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap complete.
I20260926 07:37:21.891036 7019 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:21.892052 7019 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f30b2127cb45f8bb90ba9a3888ea16, State: Initialized, Role: FOLLOWER
I20260926 07:37:21.892688 7019 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:21.892917 7019 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:37:21.893043 7019 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:37:21.893239 7019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:37:21.897099 7019 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:21.897622 7019 leader_election.cc:302] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f30b2127cb45f8bb90ba9a3888ea16; no voters:
I20260926 07:37:21.898252 7019 leader_election.cc:288] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:37:21.898386 7023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:37:21.899672 7023 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 LEADER]: Becoming Leader. State: Replica: a1f30b2127cb45f8bb90ba9a3888ea16, State: Running, Role: LEADER
I20260926 07:37:21.900290 7023 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:21.900604 7019 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:37:21.902900 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f30b2127cb45f8bb90ba9a3888ea16. Latest consensus state: current_term: 2 leader_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } } }
I20260926 07:37:21.902949 7024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } } }
I20260926 07:37:21.903299 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:37:21.903405 7024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:37:21.907251 7030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:37:21.914880 7030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:37:21.916361 7030 catalog_manager.cc:1259] Loaded cluster ID: 25c96893f57b45a8aa9ff78f62fc17c8
I20260926 07:37:21.916512 7030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:37:21.920544 7030 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:37:21.922546 7030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Loaded TSK: 0
I20260926 07:37:21.958528 7030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Generated new TSK 1
I20260926 07:37:21.958992 7030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:37:22.206446 7021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:22.206769 7021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:22.206959 7021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:22.214236 7021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:22.214341 7021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:22.214509 7021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:37:22.224294 7021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=44807
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:22.225656 7021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:22.226963 7021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:22.241454 7046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:22.241883 7047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:22.242437 7049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:22.242519 7021 server_base.cc:1034] running on GCE node
I20260926 07:37:22.243196 7021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:22.244269 7021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:22.245514 7021 hybrid_clock.cc:648] HybridClock initialized: now 1790408242245459 us; error 47 us; skew 500 ppm
I20260926 07:37:22.245865 7021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:22 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408242, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:22.249511 7021 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:37:22.251127 7021 webserver.cc:460] Webserver started at http://127.22.221.65:44807/ using document root <none> and password file <none>
I20260926 07:37:22.251909 7021 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:22.252077 7021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:22.257795 7021 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:37:22.260867 7056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:22.262337 7021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:37:22.262519 7021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "d976edb7260c4ddca6e6ee4c5475ccd3"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "5e5064763574a90c4d625bd5e44c850b"
server_key_iv: "2235b86dc444542aed5d178dec471d38"
server_key_version: "encryptionkey@0"
I20260926 07:37:22.263576 7021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:22.277407 7021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:22.281905 7021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:22.282167 7021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:22.283108 7021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:22.284536 7021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:22.284664 7021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:22.284802 7021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:22.284889 7021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:22.317471 7021 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:38743
I20260926 07:37:22.317561 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:38743 every 8 connection(s)
I20260926 07:37:22.318922 7021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:37:22.320165 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7021
I20260926 07:37:22.320806 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:41823
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:37657
--webserver_port=41809
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:37:22 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408242, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:37:22.344929 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:22.322297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:37683)
0926 07:37:22.323047 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:22.323084 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:22.324897 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:22.326337 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:22.326349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:22.326955 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:37:22.327579 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:22.327597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.329743 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.329751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:22.332019 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:22.332036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.333115 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.333125 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:22.333440 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:22.334183 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:37:22.334215 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:37:22.337022 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:37:22.340606 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:22.340616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:22.340631 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:22.341064 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:22.341525 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:22.341530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:22.341534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:22.341719 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:22.342443 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:37:22.342451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:37:22.342786 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 07:37:22.343039 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 07:37:22.343626 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":289,"threads_started":1}
I20260926 07:37:22.344921 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:22.321930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:37683 (local address 127.22.221.126:37657)
0926 07:37:22.322539 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:22.322550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:22.323262 (+ 712us) server_negotiation.cc:407] Connection header received
0926 07:37:22.325465 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:22.325477 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:22.325820 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:22.326183 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:22.327802 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.329577 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.332182 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.332875 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.333017 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:22.337314 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:22.337322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:22.337336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:22.337378 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:22.340442 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:22.341219 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:22.341226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:22.341233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:22.341393 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:22.341857 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:22.341861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:22.341863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:22.342304 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:22.342521 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:22.343311 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 07:37:22.343709 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":269,"threads_started":1}
I20260926 07:37:22.348102 7169 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37657
I20260926 07:37:22.348435 7169 heartbeater.cc:464] Registering TS with master...
I20260926 07:37:22.349175 7169 heartbeater.cc:511] Master 127.22.221.126:37657 requested a full tablet report, sending...
I20260926 07:37:22.351400 6984 ts_manager.cc:194] Registered new tserver with Master: d976edb7260c4ddca6e6ee4c5475ccd3 (127.22.221.65:38743)
I20260926 07:37:22.352885 6984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:37683
W20260926 07:37:22.655936 7173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:22.656276 7173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:22.656441 7173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:22.663652 7173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:22.663836 7173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:22.664026 7173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:37:22.674067 7173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:41823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=41809
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:22.675439 7173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:22.676746 7173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:22.690126 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:22.690204 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:22.690164 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:22.691129 7173 server_base.cc:1034] running on GCE node
I20260926 07:37:22.691721 7173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:22.692634 7173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:22.693822 7173 hybrid_clock.cc:648] HybridClock initialized: now 1790408242693755 us; error 62 us; skew 500 ppm
I20260926 07:37:22.694146 7173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:22 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408242, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:22.697227 7173 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:37:22.698674 7173 webserver.cc:460] Webserver started at http://127.22.221.66:41809/ using document root <none> and password file <none>
I20260926 07:37:22.699393 7173 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:22.699550 7173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:22.704722 7173 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:37:22.707553 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:22.708886 7173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:37:22.709065 7173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "fd2b56eeaf3f428088e7f32695f6d1c2"
server_key_iv: "77cc01eeffb7589bd3a179cf0aedfb86"
server_key_version: "encryptionkey@0"
I20260926 07:37:22.709985 7173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:22.726727 7173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:22.730463 7173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:22.730726 7173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:22.731827 7173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:22.733095 7173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:22.733227 7173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:22.733345 7173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:22.733429 7173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:22.769466 7173 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:41823
I20260926 07:37:22.769524 7300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:41823 every 8 connection(s)
I20260926 07:37:22.770905 7173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:37:22.778172 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7173
I20260926 07:37:22.778841 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:44717
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:37657
--webserver_port=38673
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:37:22 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408242, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:37:22.792215 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:22.773527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:38259 (local address 127.22.221.126:37657)
0926 07:37:22.773714 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:22.773719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:22.774913 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 07:37:22.776840 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:22.776846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:22.776942 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:22.777057 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:22.778736 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.779611 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.781419 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.781930 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.782063 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:22.785885 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:22.785893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:22.785901 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:22.785954 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:22.788836 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:22.789811 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:22.789815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:22.789818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:22.789934 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:22.790414 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:22.790417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:22.790419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:22.790680 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:22.790788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:22.791787 (+ 999us) server_negotiation.cc:299] Negotiation successful
0926 07:37:22.791966 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:37:22.793896 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:22.774003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:38259)
0926 07:37:22.774738 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:22.774774 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:22.776534 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:22.777323 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:22.777331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:22.777888 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:37:22.778553 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:22.778573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.779809 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.779813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:22.781265 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:22.781286 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:22.782172 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:22.782179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:22.782422 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:22.783175 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:37:22.783199 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:37:22.785603 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:37:22.788984 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:22.788996 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:22.789016 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:22.789681 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:22.790042 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:22.790049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:22.790056 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:22.790311 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:22.790805 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:37:22.790816 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:37:22.791386 (+ 570us) client_negotiation.cc:769] Sending connection context
0926 07:37:22.791705 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 07:37:22.792307 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":302,"threads_started":1}
I20260926 07:37:22.795524 7301 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37657
I20260926 07:37:22.795840 7301 heartbeater.cc:464] Registering TS with master...
I20260926 07:37:22.796579 7301 heartbeater.cc:511] Master 127.22.221.126:37657 requested a full tablet report, sending...
I20260926 07:37:22.798158 6984 ts_manager.cc:194] Registered new tserver with Master: d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823)
I20260926 07:37:22.798967 6984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:38259
W20260926 07:37:23.109143 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:23.109416 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:23.109607 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:23.117048 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:23.117178 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:23.117362 7304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:37:23.126963 7304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:44717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=38673
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:23.128261 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:23.129567 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:23.142518 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:23.142550 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:23.143353 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:23.144040 7304 server_base.cc:1034] running on GCE node
I20260926 07:37:23.144608 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:23.145596 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:23.146785 7304 hybrid_clock.cc:648] HybridClock initialized: now 1790408243146733 us; error 39 us; skew 500 ppm
I20260926 07:37:23.147104 7304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:23 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408243, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:23.155642 7304 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:37:23.157245 7304 webserver.cc:460] Webserver started at http://127.22.221.67:38673/ using document root <none> and password file <none>
I20260926 07:37:23.157954 7304 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:23.158070 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:23.163614 7304 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:37:23.166380 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:23.167991 7304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:37:23.168171 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "0a6f96ca0437455da919c231f906527b"
format_stamp: "Formatted at 2026-09-26 07:37:00 on dist-test-slave-qlq8"
server_key: "a397a57b0c7a6908e8ba95dd897012ac"
server_key_iv: "934935dd07a77f6d6c7e82e75d27b2f0"
server_key_version: "encryptionkey@0"
I20260926 07:37:23.169134 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:23.181279 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:23.185525 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:23.185796 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:23.186781 7304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:23.188102 7304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:23.188232 7304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:23.188369 7304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:23.188467 7304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:23.223479 7304 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:44717
I20260926 07:37:23.223560 7431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:44717 every 8 connection(s)
I20260926 07:37:23.224901 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:37:23.225533 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7304
Sep 26 07:37:23 dist-test-slave-qlq8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408243, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:37:23.242260 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.226575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.22.221.126:37657)
0926 07:37:23.226793 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.226800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.227330 (+ 530us) server_negotiation.cc:407] Connection header received
0926 07:37:23.227549 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.227555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.227646 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.227765 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:23.230398 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.231646 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.233041 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.233559 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.233693 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.236610 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:23.236617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:23.236621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:23.236659 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:23.239569 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.240235 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.240240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.240244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.240410 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.240790 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.240797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.240805 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.241083 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:23.241346 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.241659 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.241887 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":82}
I20260926 07:37:23.245728 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.227668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:56497 (local address 127.22.221.126:37657)
0926 07:37:23.230687 (+ 3019us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.230691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.230705 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:37:23.231328 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.231331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.231389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.231492 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:23.233080 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.233899 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.235461 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.235858 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.235990 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.239773 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:23.239777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:23.239780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:23.239807 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:23.242612 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.243529 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.243534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.243537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.243672 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.244105 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.244113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.244119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.244427 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:23.244570 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.245309 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.245473 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":2930,"thread_start_us":2846,"threads_started":1}
I20260926 07:37:23.246632 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.228208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:56497)
0926 07:37:23.229016 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:23.229064 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:23.231028 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:23.231748 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:23.231758 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:23.232385 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:37:23.232916 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.232932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.234061 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.234067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.235303 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.235317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.236081 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.236089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.236324 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.236939 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:37:23.236966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:37:23.239558 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:37:23.242739 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:23.242745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:23.242756 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:23.243386 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:23.243784 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:37:23.243788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:37:23.243794 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:37:23.243988 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:37:23.244550 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:37:23.244576 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:37:23.244983 (+ 407us) client_negotiation.cc:769] Sending connection context
0926 07:37:23.245209 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:37:23.245555 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":349,"threads_started":1}
I20260926 07:37:23.248005 7432 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37657
I20260926 07:37:23.248260 7432 heartbeater.cc:464] Registering TS with master...
I20260926 07:37:23.248886 7432 heartbeater.cc:511] Master 127.22.221.126:37657 requested a full tablet report, sending...
I20260926 07:37:23.250106 6984 ts_manager.cc:194] Registered new tserver with Master: 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717)
I20260926 07:37:23.250877 6984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:56497
I20260926 07:37:23.257112 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:37:23.263631 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.257966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.22.221.126:37657)
0926 07:37:23.258123 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.258127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.258589 (+ 462us) server_negotiation.cc:407] Connection header received
0926 07:37:23.258816 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.258820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.258934 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.259083 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:37:23.259576 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.260278 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.261726 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.262143 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.262275 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.262763 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:37:23.262971 (+ 208us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:37:23.263352 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50026: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 07:37:23.263762 7435 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50026: token expired
W20260926 07:37:23.264180 6908 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.221.126:37657)
I20260926 07:37:23.276575 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.264639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.22.221.126:37657)
0926 07:37:23.264789 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.264796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.264815 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:37:23.264989 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.264994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.265059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.265154 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:23.266474 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.267377 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.268707 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.269112 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.269242 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.271812 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:23.271818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:23.271822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:23.271862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:23.274399 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.275000 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.275004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.275006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.275112 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.275492 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.275495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.275497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.275756 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:23.275944 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.276183 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.276333 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:37:23.279814 6984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:37:23.288301 6984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:37:23.312446 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.301170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41823 (local address 127.0.0.1:41262)
0926 07:37:23.302466 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:23.302502 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:23.302697 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:23.303752 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:23.303764 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:23.305289 (+ 1525us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:23.305649 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.305660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.307647 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.307655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.309945 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.309957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.311516 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.311522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.311733 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.311794 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 07:37:23.311917 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:37:23.312103 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":861,"threads_started":1}
I20260926 07:37:23.313639 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.301378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.22.221.66:41823)
0926 07:37:23.301949 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.301960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.302614 (+ 654us) server_negotiation.cc:407] Connection header received
0926 07:37:23.302919 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.302929 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.303184 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.303426 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:23.305821 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.307527 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.310093 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.311205 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.312913 (+ 1708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.313114 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.313185 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.313355 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":300,"threads_started":1}
I20260926 07:37:23.314321 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.304048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.22.221.65:38743)
0926 07:37:23.306497 (+ 2449us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.306506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.306531 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:37:23.306619 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.306626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.306857 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.307068 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:23.307788 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.309338 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.311595 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.312955 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.313101 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.313289 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.313831 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.314003 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2279,"thread_start_us":2138,"threads_started":1}
I20260926 07:37:23.314342 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.303947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:38743 (local address 127.0.0.1:53944)
0926 07:37:23.304505 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:23.304527 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:23.305959 (+ 1432us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x61200002cc28. stack: 00007ff09a47066e 00007ff09a470430 00007ff0994f75ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007ff09a4660fd 00007ff09a465d1e 00007ff0ad36f3cd 00007ff0ad36f36c 00007ff0ad33bb3b 00007ff0a0e75536 00007ff0a0e73d9e 00007ff0a0efbdca 00007ff0a0ef9f68 00007ff0a0f2d82f
0926 07:37:23.305990 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:23.307067 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:23.307074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:23.307111 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:23.307557 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.307569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.309554 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.309561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.311445 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.311458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.313389 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.313393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.313555 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.313592 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:37:23.313689 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:37:23.313941 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"spinlock_wait_cycles":2771200,"thread_start_us":280,"threads_started":1}
I20260926 07:37:23.316826 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.302807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:44717 (local address 127.0.0.1:36966)
0926 07:37:23.303715 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:23.303740 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:23.305133 (+ 1393us) spinlock_profiling.cc:243] Waited 916 us on lock 0x61200002cc28. stack: 00007ff09a47066e 00007ff09a470430 00007ff0994f75ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007ff09a4660fd 00007ff09a465d1e 00007ff0ad36f3cd 00007ff0ad36f36c 00007ff0ad33bb3b 00007ff0a0e75536 00007ff0a0e73d9e 00007ff0a0efbdca 00007ff0a0ef9f68 00007ff0a0f2d82f
0926 07:37:23.305163 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:23.308254 (+ 3091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:23.308261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:23.308288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:23.308734 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.308746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.310954 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.310961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.313132 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.313147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.315775 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.315781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.315963 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.316000 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:37:23.316098 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:37:23.316233 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"spinlock_wait_cycles":2107136,"thread_start_us":646,"threads_started":1}
I20260926 07:37:23.317586 7236 tablet_service.cc:1467] Processing CreateTablet for tablet c849464aff9847ddbe25274f77403cea (DEFAULT_TABLE table=test-table [id=c104eda6678344b8bbe4faa7b1841ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:37:23.318571 7104 tablet_service.cc:1467] Processing CreateTablet for tablet c849464aff9847ddbe25274f77403cea (DEFAULT_TABLE table=test-table [id=c104eda6678344b8bbe4faa7b1841ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:37:23.319023 7236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c849464aff9847ddbe25274f77403cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:37:23.319761 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c849464aff9847ddbe25274f77403cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:37:23.316654 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.303165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.22.221.67:44717)
0926 07:37:23.307731 (+ 4566us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.307738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.307765 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:37:23.307844 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.307848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.308036 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.308927 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:23.309192 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.310737 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.313349 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.315600 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.315730 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.315934 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.316157 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.316300 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4398,"thread_start_us":4220,"threads_started":1}
I20260926 07:37:23.324174 7367 tablet_service.cc:1467] Processing CreateTablet for tablet c849464aff9847ddbe25274f77403cea (DEFAULT_TABLE table=test-table [id=c104eda6678344b8bbe4faa7b1841ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:37:23.325513 7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c849464aff9847ddbe25274f77403cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:37:23.332713 7448 tablet_bootstrap.cc:492] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Bootstrap starting.
I20260926 07:37:23.335763 7448 tablet_bootstrap.cc:654] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Neither blocks nor log segments found. Creating new log.
I20260926 07:37:23.337021 7448 log.cc:824] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:23.339499 7449 tablet_bootstrap.cc:492] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Bootstrap starting.
I20260926 07:37:23.340024 7448 tablet_bootstrap.cc:492] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: No bootstrap required, opened a new log
I20260926 07:37:23.340294 7448 ts_tablet_manager.cc:1397] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 07:37:23.340732 7451 tablet_bootstrap.cc:492] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Bootstrap starting.
I20260926 07:37:23.342509 7449 tablet_bootstrap.cc:654] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Neither blocks nor log segments found. Creating new log.
I20260926 07:37:23.343611 7449 log.cc:824] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:23.344316 7451 tablet_bootstrap.cc:654] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Neither blocks nor log segments found. Creating new log.
I20260926 07:37:23.345475 7451 log.cc:824] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:23.346323 7449 tablet_bootstrap.cc:492] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: No bootstrap required, opened a new log
I20260926 07:37:23.346601 7449 ts_tablet_manager.cc:1397] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 07:37:23.348399 7451 tablet_bootstrap.cc:492] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: No bootstrap required, opened a new log
I20260926 07:37:23.348637 7451 ts_tablet_manager.cc:1397] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:37:23.350499 7448 raft_consensus.cc:348] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.350925 7448 raft_consensus.cc:374] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:37:23.351023 7448 raft_consensus.cc:729] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b33f5f3f6c4abe9c991df6f68cedfd, State: Initialized, Role: FOLLOWER
I20260926 07:37:23.351671 7448 consensus_queue.cc:260] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.352769 7301 heartbeater.cc:503] Master 127.22.221.126:37657 was elected leader, sending a full tablet report...
I20260926 07:37:23.353425 7448 ts_tablet_manager.cc:1428] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260926 07:37:23.354840 7449 raft_consensus.cc:348] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.355165 7449 raft_consensus.cc:374] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:37:23.355320 7449 raft_consensus.cc:729] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d976edb7260c4ddca6e6ee4c5475ccd3, State: Initialized, Role: FOLLOWER
I20260926 07:37:23.355926 7169 heartbeater.cc:503] Master 127.22.221.126:37657 was elected leader, sending a full tablet report...
I20260926 07:37:23.355952 7449 consensus_queue.cc:260] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.359877 7451 raft_consensus.cc:348] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.360323 7451 raft_consensus.cc:374] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:37:23.360441 7451 raft_consensus.cc:729] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6f96ca0437455da919c231f906527b, State: Initialized, Role: FOLLOWER
I20260926 07:37:23.360797 7449 ts_tablet_manager.cc:1428] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260926 07:37:23.361060 7451 consensus_queue.cc:260] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.362121 7432 heartbeater.cc:503] Master 127.22.221.126:37657 was elected leader, sending a full tablet report...
I20260926 07:37:23.362746 7451 ts_tablet_manager.cc:1428] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 07:37:23.433301 7455 raft_consensus.cc:482] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:37:23.433641 7455 raft_consensus.cc:504] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.435361 7455 leader_election.cc:288] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823), 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717)
I20260926 07:37:23.441846 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.435731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:42017 (local address 127.22.221.66:41823)
0926 07:37:23.435996 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.436004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.436347 (+ 343us) server_negotiation.cc:407] Connection header received
0926 07:37:23.436602 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.436611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.436718 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.436797 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:23.437580 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.438297 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.440172 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.440919 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.441113 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.441187 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.441460 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.441588 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:37:23.442137 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.435603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41823 (local address 127.22.221.65:42017)
0926 07:37:23.436216 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:23.436234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:23.436384 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:23.437032 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:23.437038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:23.437080 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:23.437422 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.437429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.438452 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.438464 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.440006 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.440018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.441128 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.441134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.441305 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.441329 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:37:23.441434 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:37:23.441574 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":416,"threads_started":1}
I20260926 07:37:23.442351 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.436734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:37465 (local address 127.22.221.67:44717)
0926 07:37:23.436953 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.436960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.437271 (+ 311us) server_negotiation.cc:407] Connection header received
0926 07:37:23.437490 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.437493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.437552 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.437649 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:37:23.438148 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.438873 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.440645 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.441409 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.441595 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.441682 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.441982 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.442102 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:37:23.442402 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.436411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:44717 (local address 127.22.221.65:37465)
0926 07:37:23.437145 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:23.437159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:37:23.437338 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:37:23.437663 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:37:23.437668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:37:23.437691 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:37:23.438006 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.438014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.439007 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.439014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.440484 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:37:23.440495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.441625 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.441631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:37:23.441812 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.441834 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:37:23.441942 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:37:23.442061 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":417,"threads_started":1}
I20260926 07:37:23.443492 7256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849464aff9847ddbe25274f77403cea" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" is_pre_election: true
I20260926 07:37:23.443631 7387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849464aff9847ddbe25274f77403cea" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6f96ca0437455da919c231f906527b" is_pre_election: true
I20260926 07:37:23.444079 7256 raft_consensus.cc:2454] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 0.
I20260926 07:37:23.444129 7387 raft_consensus.cc:2454] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 0.
I20260926 07:37:23.444773 7060 leader_election.cc:302] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2b33f5f3f6c4abe9c991df6f68cedfd, d976edb7260c4ddca6e6ee4c5475ccd3; no voters:
I20260926 07:37:23.445138 7455 raft_consensus.cc:2781] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:37:23.445253 7455 raft_consensus.cc:482] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:37:23.445395 7455 raft_consensus.cc:3037] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:37:23.447765 7455 raft_consensus.cc:504] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.448316 7455 leader_election.cc:288] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 election: Requested vote from peers d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823), 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717)
I20260926 07:37:23.448746 7256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849464aff9847ddbe25274f77403cea" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
I20260926 07:37:23.448945 7256 raft_consensus.cc:3037] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:37:23.449160 7387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849464aff9847ddbe25274f77403cea" candidate_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6f96ca0437455da919c231f906527b"
I20260926 07:37:23.449371 7387 raft_consensus.cc:3037] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:37:23.452226 7256 raft_consensus.cc:2454] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 1.
I20260926 07:37:23.452567 7387 raft_consensus.cc:2454] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d976edb7260c4ddca6e6ee4c5475ccd3 in term 1.
I20260926 07:37:23.452621 7060 leader_election.cc:302] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2b33f5f3f6c4abe9c991df6f68cedfd, d976edb7260c4ddca6e6ee4c5475ccd3; no voters:
I20260926 07:37:23.452927 7455 raft_consensus.cc:2781] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:37:23.453434 7455 raft_consensus.cc:686] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 LEADER]: Becoming Leader. State: Replica: d976edb7260c4ddca6e6ee4c5475ccd3, State: Running, Role: LEADER
I20260926 07:37:23.454010 7455 consensus_queue.cc:237] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } }
I20260926 07:37:23.458627 6984 catalog_manager.cc:5495] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d976edb7260c4ddca6e6ee4c5475ccd3 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d976edb7260c4ddca6e6ee4c5475ccd3" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 38743 } health_report { overall_health: HEALTHY } } }
W20260926 07:37:23.476670 7433 tablet.cc:2369] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:37:23.507529 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.501837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.22.221.65:38743)
0926 07:37:23.502044 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.502052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.502077 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:37:23.502358 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.502362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.502452 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.502606 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:37:23.503000 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.503825 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.505181 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.505634 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.505788 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.506348 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:37:23.506870 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:37:23.507038 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.507097 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.507181 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:37:23.516157 7387 raft_consensus.cc:1261] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Refusing update from remote peer d976edb7260c4ddca6e6ee4c5475ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:37:23.516157 7256 raft_consensus.cc:1261] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Refusing update from remote peer d976edb7260c4ddca6e6ee4c5475ccd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:37:23.517222 7459 consensus_queue.cc:1035] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:37:23.517650 7455 consensus_queue.cc:1035] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a6f96ca0437455da919c231f906527b" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:37:23.522940 7302 tablet.cc:2369] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:37:23.534070 7463 mvcc.cc:205] Tried to move back new op lower bound from 7333512165431132160 to 7333512165191593984. Current Snapshot: MvccSnapshot[applied={T|T < 7333512165431132160}]
I20260926 07:37:23.544055 6983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:37:23.544306 6983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:37:23.547842 6983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Sending DeleteTablet for 3 replicas of tablet c849464aff9847ddbe25274f77403cea
I20260926 07:37:23.548913 7236 tablet_service.cc:1514] Processing DeleteTablet for tablet c849464aff9847ddbe25274f77403cea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:37:23 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:41262
I20260926 07:37:23.549153 7367 tablet_service.cc:1514] Processing DeleteTablet for tablet c849464aff9847ddbe25274f77403cea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:37:23 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:36966
I20260926 07:37:23.549373 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet c849464aff9847ddbe25274f77403cea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:37:23 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:53944
I20260926 07:37:23.550549 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 7021
I20260926 07:37:23.560065 7474 tablet_replica.cc:330] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: stopping tablet replica
I20260926 07:37:23.561108 7472 tablet_replica.cc:330] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: stopping tablet replica
I20260926 07:37:23.561101 7473 tablet_replica.cc:330] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: stopping tablet replica
I20260926 07:37:23.561503 7472 raft_consensus.cc:2229] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:37:23.561705 7473 raft_consensus.cc:2229] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:37:23.562011 7472 raft_consensus.cc:2258] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:37:23.562273 7473 raft_consensus.cc:2258] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:37:23.563097 7474 raft_consensus.cc:2229] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:37:23.563867 7474 raft_consensus.cc:2258] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:37:23.565843 7474 ts_tablet_manager.cc:1905] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:37:23.565896 7473 ts_tablet_manager.cc:1905] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:37:23.572211 7472 ts_tablet_manager.cc:1905] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:37:23.574018 7474 ts_tablet_manager.cc:1918] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:37:23.574240 7474 log.cc:1196] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/c849464aff9847ddbe25274f77403cea
I20260926 07:37:23.574884 7474 ts_tablet_manager.cc:1939] T c849464aff9847ddbe25274f77403cea P d976edb7260c4ddca6e6ee4c5475ccd3: Deleting consensus metadata
I20260926 07:37:23.576202 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.552322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.22.221.65:38743)
0926 07:37:23.552542 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.552550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.552572 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:37:23.552637 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.552644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.552746 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.552858 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:23.555322 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.556971 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.560177 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.563379 (+ 3202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.563504 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.567262 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:23.567280 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:23.567298 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:23.567352 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:23.571796 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.572715 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.572724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.572728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.572929 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.573400 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.573407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.573425 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.573865 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:23.575360 (+ 1495us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.575529 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.575773 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:37:23.576860 7164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:23.577337 6971 catalog_manager.cc:4841] TS d976edb7260c4ddca6e6ee4c5475ccd3 (127.22.221.65:38743): tablet c849464aff9847ddbe25274f77403cea (table test-table [id=c104eda6678344b8bbe4faa7b1841ba6]) successfully deleted
I20260926 07:37:23.579010 7473 ts_tablet_manager.cc:1918] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:37:23.579211 7473 log.cc:1196] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/c849464aff9847ddbe25274f77403cea
I20260926 07:37:23.579909 7473 ts_tablet_manager.cc:1939] T c849464aff9847ddbe25274f77403cea P 0a6f96ca0437455da919c231f906527b: Deleting consensus metadata
I20260926 07:37:23.579947 7472 ts_tablet_manager.cc:1918] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:37:23.580135 7472 log.cc:1196] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/c849464aff9847ddbe25274f77403cea
I20260926 07:37:23.580916 7472 ts_tablet_manager.cc:1939] T c849464aff9847ddbe25274f77403cea P d2b33f5f3f6c4abe9c991df6f68cedfd: Deleting consensus metadata
I20260926 07:37:23.582002 6971 catalog_manager.cc:4841] TS 0a6f96ca0437455da919c231f906527b (127.22.221.67:44717): tablet c849464aff9847ddbe25274f77403cea (table test-table [id=c104eda6678344b8bbe4faa7b1841ba6]) successfully deleted
I20260926 07:37:23.582556 6971 catalog_manager.cc:4841] TS d2b33f5f3f6c4abe9c991df6f68cedfd (127.22.221.66:41823): tablet c849464aff9847ddbe25274f77403cea (table test-table [id=c104eda6678344b8bbe4faa7b1841ba6]) successfully deleted
I20260926 07:37:23.673252 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 7021
I20260926 07:37:23.692602 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 7173
I20260926 07:37:23.705883 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.693456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.22.221.66:41823)
0926 07:37:23.693630 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.693634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.693651 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:37:23.693784 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.693788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.693850 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.693931 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:23.695224 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.696017 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.697266 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.697666 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.697757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.700389 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:23.700402 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:23.700417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:23.700458 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:23.703541 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.704184 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.704190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.704192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.704311 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.704611 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.704614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.704616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.704956 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:23.705173 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.705482 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.705635 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:37:23.706569 7296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:23.797106 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 7173
I20260926 07:37:23.814255 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 7304
I20260926 07:37:23.829013 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:23.815081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.22.221.67:44717)
0926 07:37:23.815322 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 07:37:23.815327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:37:23.815522 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:37:23.815670 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:37:23.815674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:37:23.815741 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:37:23.815828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:37:23.817339 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.818068 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.819839 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:37:23.820299 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:37:23.820455 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:37:23.823204 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:37:23.823307 (+ 103us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:37:23.823328 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:37:23.823373 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:37:23.826097 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.826855 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.826863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.826865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.827004 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:37:23.827457 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:37:23.827461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:37:23.827463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:37:23.827814 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:37:23.828107 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 07:37:23.828440 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:37:23.828680 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:37:23.829680 7427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:23.922773 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 7304
I20260926 07:37:23.939519 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 6951
I20260926 07:37:23.940706 7014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:37:24.057546 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 6951
I20260926 07:37:44.079159 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37657
--webserver_interface=127.22.221.126
--webserver_port=36895
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:37:44.407948 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:44.408242 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:44.408330 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:44.415392 7479 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:37:44.415485 7479 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:37:44.415524 7479 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:37:44.415567 7479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:37:44.415619 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:44.415688 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:44.415733 7479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:37:44.415786 7479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:37:44.425524 7479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=36895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:44.426796 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:44.428087 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:44.440012 7485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:44.440068 7484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:44.440092 7487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:44.440976 7479 server_base.cc:1034] running on GCE node
I20260926 07:37:44.441666 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:44.442957 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:44.444166 7479 hybrid_clock.cc:648] HybridClock initialized: now 1790408264444080 us; error 66 us; skew 500 ppm
I20260926 07:37:44.444532 7479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:44 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:44.454588 7479 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:37:44.456223 7479 webserver.cc:460] Webserver started at http://127.22.221.126:36895/ using document root <none> and password file <none>
I20260926 07:37:44.456923 7479 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:44.457022 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:44.462960 7479 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 07:37:44.467196 7494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:44.469142 7479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:37:44.469379 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "a1f30b2127cb45f8bb90ba9a3888ea16"
format_stamp: "Formatted at 2026-09-26 07:36:58 on dist-test-slave-qlq8"
server_key: "80d2c334165b534d68b9b3cd7225a143"
server_key_iv: "8b59e91ac2b8b89a2420f880fa52cf5b"
server_key_version: "encryptionkey@0"
I20260926 07:37:44.470474 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:44.489131 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:44.493242 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:44.493542 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:44.511559 7479 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37657
I20260926 07:37:44.511591 7545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37657 every 8 connection(s)
I20260926 07:37:44.513062 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:37:44.513669 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7479
I20260926 07:37:44.514652 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:38743
--local_ip_for_outbound_sockets=127.22.221.65
--tserver_master_addrs=127.22.221.126:37657
--webserver_port=44807
--webserver_interface=127.22.221.65
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:37:44.520416 7546 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:37:44.523658 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap starting.
I20260926 07:37:44.543773 7546 log.cc:824] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Log is configured to *not* fsync() on all Append() calls
I20260926 07:37:44.559274 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:37:44.559837 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Bootstrap complete.
I20260926 07:37:44.569164 7546 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:44.570086 7546 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f30b2127cb45f8bb90ba9a3888ea16, State: Initialized, Role: FOLLOWER
I20260926 07:37:44.570838 7546 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:44.571121 7546 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:37:44.571274 7546 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:37:44.571465 7546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:37:44.575510 7546 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:44.575989 7546 leader_election.cc:302] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f30b2127cb45f8bb90ba9a3888ea16; no voters:
I20260926 07:37:44.576678 7546 leader_election.cc:288] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:37:44.576828 7550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:37:44.579028 7546 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:37:44.583288 7550 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [term 3 LEADER]: Becoming Leader. State: Replica: a1f30b2127cb45f8bb90ba9a3888ea16, State: Running, Role: LEADER
I20260926 07:37:44.584021 7550 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } }
I20260926 07:37:44.587294 7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f30b2127cb45f8bb90ba9a3888ea16. Latest consensus state: current_term: 3 leader_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } } }
I20260926 07:37:44.587260 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f30b2127cb45f8bb90ba9a3888ea16" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37657 } } }
I20260926 07:37:44.587654 7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:37:44.587703 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:37:44.599428 7564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:37:44.605914 7564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:37:44.606969 7564 catalog_manager.cc:1259] Loaded cluster ID: 25c96893f57b45a8aa9ff78f62fc17c8
I20260926 07:37:44.607074 7564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:37:44.609773 7564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:37:44.610884 7564 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Loaded TSK: 0
I20260926 07:37:44.611016 7564 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1f30b2127cb45f8bb90ba9a3888ea16: Loaded TSK: 1
F20260926 07:37:44.612025 7564 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1304a899ad google::LogMessage::Fail() at ??:0
@ 0x7f1304a8c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1304a894de google::LogMessage::Flush() at ??:0
@ 0x7f1304a8aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f13087ed7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f13087dc73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f131a4b81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f131a4b7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f131a5432a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f131a5430c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f131a58d5e9 std::function<>::operator()() at ??:0
@ 0x7f131a4bbcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f131a4b9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f131a541610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f131a541439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f13076d5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f13076d891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f13076d8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f13076b3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f130a98d6db start_thread at ??:0
@ 0x7f130392361f clone at ??:0
W20260926 07:37:44.881641 7548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:44.881963 7548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:44.882123 7548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:44.889626 7548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:44.889783 7548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:44.889961 7548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:37:44.900519 7548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=44807
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:44.901897 7548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:44.903379 7548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:44.916733 7593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:44.916765 7594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:44.916797 7596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:44.917601 7548 server_base.cc:1034] running on GCE node
I20260926 07:37:44.918192 7548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:44.919127 7548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:44.920323 7548 hybrid_clock.cc:648] HybridClock initialized: now 1790408264920262 us; error 61 us; skew 500 ppm
I20260926 07:37:44.920643 7548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:44 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:44.923929 7548 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:37:44.925374 7548 webserver.cc:460] Webserver started at http://127.22.221.65:44807/ using document root <none> and password file <none>
I20260926 07:37:44.926050 7548 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:44.926180 7548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:44.931710 7548 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:37:44.934396 7603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:44.935923 7548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:37:44.936106 7548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "d976edb7260c4ddca6e6ee4c5475ccd3"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "5e5064763574a90c4d625bd5e44c850b"
server_key_iv: "2235b86dc444542aed5d178dec471d38"
server_key_version: "encryptionkey@0"
I20260926 07:37:44.937085 7548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:44.955101 7548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:44.959568 7548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:44.959844 7548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:44.960865 7548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:44.962244 7548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:44.962373 7548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:44.962496 7548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:44.962590 7548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:44.996265 7548 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:38743
I20260926 07:37:44.996364 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:38743 every 8 connection(s)
I20260926 07:37:44.997860 7548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:37:45.001228 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7548
I20260926 07:37:45.001878 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:41823
--local_ip_for_outbound_sockets=127.22.221.66
--tserver_master_addrs=127.22.221.126:37657
--webserver_port=41809
--webserver_interface=127.22.221.66
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:37:45.003105 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.001001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:50537)
0926 07:37:45.001865 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.002164 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":366,"threads_started":1}
W20260926 07:37:45.003940 7716 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:37657 (0 consecutive failures): Network error: Failed to ping master at 127.22.221.126:37657: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
I20260926 07:37:45.005012 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.004578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:34701)
0926 07:37:45.004797 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.004882 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 07:37:45.006048 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.005657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:47039)
0926 07:37:45.005848 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.005932 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 07:37:45.006374 7716 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:37:45.339143 7719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:45.339501 7719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:45.339680 7719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:45.346890 7719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:45.347009 7719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:45.347257 7719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:37:45.356787 7719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:41823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=41809
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:45.358044 7719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:45.359365 7719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:45.372498 7724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:45.372601 7727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:45.372576 7725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:45.376973 7719 server_base.cc:1034] running on GCE node
I20260926 07:37:45.377760 7719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:45.378769 7719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:45.380034 7719 hybrid_clock.cc:648] HybridClock initialized: now 1790408265379986 us; error 44 us; skew 500 ppm
I20260926 07:37:45.380424 7719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:45 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:45.383787 7719 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:37:45.385424 7719 webserver.cc:460] Webserver started at http://127.22.221.66:41809/ using document root <none> and password file <none>
I20260926 07:37:45.386150 7719 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:45.386300 7719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:45.391851 7719 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:37:45.394660 7734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:45.396054 7719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:37:45.396258 7719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "d2b33f5f3f6c4abe9c991df6f68cedfd"
format_stamp: "Formatted at 2026-09-26 07:36:59 on dist-test-slave-qlq8"
server_key: "fd2b56eeaf3f428088e7f32695f6d1c2"
server_key_iv: "77cc01eeffb7589bd3a179cf0aedfb86"
server_key_version: "encryptionkey@0"
I20260926 07:37:45.397324 7719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:45.413010 7719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:45.417443 7719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:45.417711 7719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:45.418717 7719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:45.420099 7719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:45.420223 7719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:45.420356 7719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:45.420444 7719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:45.454952 7719 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:41823
I20260926 07:37:45.455006 7846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:41823 every 8 connection(s)
I20260926 07:37:45.456475 7719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:37:45.458170 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7719
I20260926 07:37:45.458976 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:44717
--local_ip_for_outbound_sockets=127.22.221.67
--tserver_master_addrs=127.22.221.126:37657
--webserver_port=38673
--webserver_interface=127.22.221.67
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:37:45.461719 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.459782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:35855)
0926 07:37:45.460576 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.460838 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":324,"threads_started":1}
W20260926 07:37:45.462366 7847 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:37657 (0 consecutive failures): Network error: Failed to ping master at 127.22.221.126:37657: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
I20260926 07:37:45.463442 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.462978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:50935)
0926 07:37:45.463163 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.463324 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 07:37:45.464329 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.463989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:52491)
0926 07:37:45.464129 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.464217 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:37:45.464571 7847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:37:45.777801 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:37:45.778090 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:37:45.778224 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:37:45.785526 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:37:45.785634 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:37:45.785821 7850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:37:45.795465 7850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:44717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=38673
--tserver_master_addrs=127.22.221.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:37:45.796757 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:37:45.798053 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:37:45.811201 7855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:45.811162 7856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:37:45.811530 7858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:37:45.812044 7850 server_base.cc:1034] running on GCE node
I20260926 07:37:45.812731 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:37:45.813719 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:37:45.815002 7850 hybrid_clock.cc:648] HybridClock initialized: now 1790408265814924 us; error 63 us; skew 500 ppm
I20260926 07:37:45.815461 7850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:37:45 dist-test-slave-qlq8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:37:45.818789 7850 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:37:45.820326 7850 webserver.cc:460] Webserver started at http://127.22.221.67:38673/ using document root <none> and password file <none>
I20260926 07:37:45.821033 7850 fs_manager.cc:359] Metadata directory not provided
I20260926 07:37:45.821177 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:37:45.826629 7850 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:37:45.829296 7865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:45.830582 7850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:37:45.830747 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "0a6f96ca0437455da919c231f906527b"
format_stamp: "Formatted at 2026-09-26 07:37:00 on dist-test-slave-qlq8"
server_key: "a397a57b0c7a6908e8ba95dd897012ac"
server_key_iv: "934935dd07a77f6d6c7e82e75d27b2f0"
server_key_version: "encryptionkey@0"
I20260926 07:37:45.831797 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:37:45.846979 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:37:45.851122 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:37:45.851413 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:37:45.852317 7850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:37:45.853658 7850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:37:45.853777 7850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:45.853893 7850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:37:45.853981 7850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:37:45.884917 7850 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:44717
I20260926 07:37:45.884968 7977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:44717 every 8 connection(s)
I20260926 07:37:45.886353 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:37:45.891199 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.889349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:55425)
0926 07:37:45.890075 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.890310 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":304,"threads_started":1}
W20260926 07:37:45.891875 7978 heartbeater.cc:644] Failed to heartbeat to 127.22.221.126:37657 (0 consecutive failures): Network error: Failed to ping master at 127.22.221.126:37657: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
I20260926 07:37:45.892637 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.892346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:40691)
0926 07:37:45.892474 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.892539 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:37:45.893420 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:45.893115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:57579)
0926 07:37:45.893249 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:45.893319 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:37:45.893661 7978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:37:45.893702 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 7850
I20260926 07:37:46.007999 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:46.006980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:47107)
0926 07:37:46.007539 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:46.007692 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
I20260926 07:37:46.466445 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:46.465351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:52061)
0926 07:37:46.465970 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:46.466142 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":311,"threads_started":1}
I20260926 07:37:46.895326 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:46.894400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:38955)
0926 07:37:46.894898 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:46.895011 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260926 07:37:47.009661 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:47.008958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:44025)
0926 07:37:47.009412 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:47.009499 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
I20260926 07:37:47.468232 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:47.467528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:40165)
0926 07:37:47.467979 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:47.468068 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":218,"threads_started":1}
I20260926 07:37:47.896953 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:47.896227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:32865)
0926 07:37:47.896701 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:47.896785 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260926 07:37:48.011514 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:48.010483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:44371)
0926 07:37:48.010943 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:48.011324 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":272,"threads_started":1}
I20260926 07:37:48.470285 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:48.469305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:47363)
0926 07:37:48.469907 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:48.470050 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":281,"threads_started":1}
I20260926 07:37:48.898540 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:48.897837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:35859)
0926 07:37:48.898297 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:48.898379 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":232,"threads_started":1}
I20260926 07:37:49.013105 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:49.012398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:32949)
0926 07:37:49.012830 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:49.012913 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":203,"threads_started":1}
I20260926 07:37:49.472137 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:49.471217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:42389)
0926 07:37:49.471736 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:49.471884 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260926 07:37:49.900400 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:49.899486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:49713)
0926 07:37:49.900006 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:49.900104 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":241,"threads_started":1}
I20260926 07:37:50.014801 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:50.014012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:43039)
0926 07:37:50.014537 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:50.014623 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":288,"threads_started":1}
I20260926 07:37:50.473935 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:50.473125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:32933)
0926 07:37:50.473666 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:50.473761 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":297,"threads_started":1}
I20260926 07:37:50.902465 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:50.901566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:57869)
0926 07:37:50.902182 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:50.902281 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":367,"threads_started":1}
I20260926 07:37:51.016635 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:51.015925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:53597)
0926 07:37:51.016376 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:51.016473 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":232,"threads_started":1}
I20260926 07:37:51.475780 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:51.474982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:32811)
0926 07:37:51.475511 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:51.475606 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":306,"threads_started":1}
I20260926 07:37:51.904328 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:51.903582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:52487)
0926 07:37:51.904071 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:51.904167 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260926 07:37:52.018499 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:52.017673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:47007)
0926 07:37:52.018152 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:52.018274 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":199,"threads_started":1}
I20260926 07:37:52.477531 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:52.476682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:37755)
0926 07:37:52.477208 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:52.477301 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":217,"threads_started":1}
I20260926 07:37:52.905951 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:52.905216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:37743)
0926 07:37:52.905660 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:52.905742 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":203,"threads_started":1}
I20260926 07:37:53.020288 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:53.019560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:55511)
0926 07:37:53.020041 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:53.020127 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
I20260926 07:37:53.479444 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:53.478494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:39475)
0926 07:37:53.479064 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:53.479160 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":313,"threads_started":1}
I20260926 07:37:53.907630 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:53.906888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:39459)
0926 07:37:53.907371 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:53.907458 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":224,"threads_started":1}
I20260926 07:37:54.022039 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:54.021223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:41485)
0926 07:37:54.021755 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:54.021853 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":255,"threads_started":1}
I20260926 07:37:54.481302 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:54.480499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:45153)
0926 07:37:54.481011 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:54.481113 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260926 07:37:54.909286 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:54.908549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:51939)
0926 07:37:54.909017 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:54.909103 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260926 07:37:55.023798 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:55.023088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:46497)
0926 07:37:55.023544 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:55.023639 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260926 07:37:55.482887 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:55.482224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:43341)
0926 07:37:55.482649 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:55.482728 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260926 07:37:55.910926 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:55.910236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:55881)
0926 07:37:55.910680 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:55.910767 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":233,"threads_started":1}
I20260926 07:37:56.025477 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:56.024749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:55369)
0926 07:37:56.025207 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:56.025303 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260926 07:37:56.484723 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:56.483894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:41963)
0926 07:37:56.484445 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:56.484540 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":325,"threads_started":1}
I20260926 07:37:56.912765 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:56.911974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:60519)
0926 07:37:56.912493 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:56.912589 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":288,"threads_started":1}
I20260926 07:37:57.027196 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:57.026517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:50385)
0926 07:37:57.026957 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:57.027037 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260926 07:37:57.496007 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:57.485794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:50527)
0926 07:37:57.495702 (+ 9908us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:57.495844 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9764,"thread_start_us":9653,"threads_started":1}
I20260926 07:37:57.914532 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:57.913741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:44107)
0926 07:37:57.914270 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:57.914359 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":277,"threads_started":1}
I20260926 07:37:58.029053 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:58.028211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:51851)
0926 07:37:58.028765 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:58.028862 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":301,"threads_started":1}
I20260926 07:37:58.497855 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:58.497010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:36685)
0926 07:37:58.497567 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:58.497667 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":258,"threads_started":1}
I20260926 07:37:58.916452 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:58.915518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:52451)
0926 07:37:58.916098 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:58.916224 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260926 07:37:59.030691 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:59.029942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:50239)
0926 07:37:59.030388 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:59.030469 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260926 07:37:59.499756 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:59.498822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:52937)
0926 07:37:59.499404 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:59.499535 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":275,"threads_started":1}
I20260926 07:37:59.918087 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:37:59.917349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:56231)
0926 07:37:59.917767 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 07:37:59.917849 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":224,"threads_started":1}
I20260926 07:38:00.032851 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:00.031979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.65:37101)
0926 07:38:00.032560 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:00.032666 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":318,"threads_started":1}
I20260926 07:38:00.501780 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:00.500932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.66:43409)
0926 07:38:00.501490 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:00.501590 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":330,"threads_started":1}
I20260926 07:38:00.901048 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 7548
I20260926 07:38:00.919389 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:00.902220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.22.221.65:38743)
0926 07:38:00.902835 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:00.902849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:00.902882 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:38:00.904500 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:00.904509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:00.904939 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:00.905223 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:00.906654 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:00.907968 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:00.909236 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:00.909914 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:00.910060 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:00.912764 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:00.912771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:00.912786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:00.912823 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:00.916027 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:00.916671 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:00.916678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:00.916684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:00.916884 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:00.917298 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:00.917302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:00.917304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:00.917835 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:00.918062 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:00.918465 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 07:38:00.918875 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":314,"threads_started":1}
I20260926 07:38:00.920014 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:00.919196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37657 (local address 127.22.221.67:50717)
0926 07:38:00.919706 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:00.919815 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":214,"threads_started":1}
I20260926 07:38:00.920912 7711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:01.005033 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 7548
I20260926 07:38:01.022012 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 7719
I20260926 07:38:01.039965 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:01.022926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.22.221.66:41823)
0926 07:38:01.023547 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:01.023560 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:01.023596 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:38:01.025138 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:01.025148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:01.025574 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:01.025867 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:01.027324 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:01.028530 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:01.029738 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:01.030516 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:01.030650 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:01.033499 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:01.033506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:01.033521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:01.033564 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:01.036768 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:01.037445 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:01.037452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:01.037458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:01.037639 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:01.037974 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:01.037977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:01.037979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:01.038496 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:01.038744 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:01.039126 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 07:38:01.039680 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":362,"threads_started":1}
I20260926 07:38:01.041299 7842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:01.123811 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 7719
I20260926 07:38:01.141109 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 7850
I20260926 07:38:01.158460 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:01.142102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.22.221.67:44717)
0926 07:38:01.142675 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:01.142688 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:01.142726 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:38:01.144393 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:01.144402 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:01.144833 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:01.145126 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:01.146444 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:01.147783 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:01.149006 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:01.149654 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:01.149783 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:01.152359 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:01.152366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:01.152381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:01.152415 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:01.155389 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:01.156096 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:01.156102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:01.156109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:01.156268 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:01.156627 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:01.156630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:01.156632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:01.157117 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:01.157334 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:01.157738 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 07:38:01.158163 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":237,"threads_started":1}
I20260926 07:38:01.159981 7973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:01.244226 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 7850
2026-09-26T07:38:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64465 ms)
[----------] 30 tests from SecurityITest (292193 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:01 dist-test-slave-qlq8 krb5kdc[8035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:01 dist-test-slave-qlq8 krb5kdc[8035](info): set up 2 sockets
Sep 26 07:38:01 dist-test-slave-qlq8 krb5kdc[8035](info): commencing operation
krb5kdc: starting...
W20260926 07:38:02.355597 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:02 dist-test-slave-qlq8 krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:02.615353 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35335
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:02.943981 8048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:02.944273 8048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:02.944362 8048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:02.951617 8048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:02.951728 8048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:02.951774 8048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:02.951834 8048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:02.951877 8048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:02.961628 8048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:02.962894 8048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:02.964179 8048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:02.976020 8053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:02.976076 8054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:02.975975 8056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:02.976944 8048 server_base.cc:1034] running on GCE node
I20260926 07:38:02.977819 8048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:02.979604 8048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:02.980855 8048 hybrid_clock.cc:648] HybridClock initialized: now 1790408282980804 us; error 44 us; skew 500 ppm
I20260926 07:38:02.981262 8048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:02 dist-test-slave-qlq8 krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408282, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:02.984716 8048 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:02.986509 8048 webserver.cc:460] Webserver started at http://127.22.221.126:35759/ using document root <none> and password file <none>
I20260926 07:38:02.987217 8048 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:02.987417 8048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:02.987686 8048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:02.990731 8048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "d38a6d5c2145419e8f8aa1d116f69b85"
format_stamp: "Formatted at 2026-09-26 07:38:02 on dist-test-slave-qlq8"
server_key: "87358093749e93434f9c3dd0c16e2329"
server_key_iv: "f16a1cd35b172fa2c23fe1777c6c9ba4"
server_key_version: "encryptionkey@0"
I20260926 07:38:02.991575 8048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "d38a6d5c2145419e8f8aa1d116f69b85"
format_stamp: "Formatted at 2026-09-26 07:38:02 on dist-test-slave-qlq8"
server_key: "87358093749e93434f9c3dd0c16e2329"
server_key_iv: "f16a1cd35b172fa2c23fe1777c6c9ba4"
server_key_version: "encryptionkey@0"
I20260926 07:38:02.996861 8048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:38:03.000520 8063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:03.002038 8048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:38:03.002269 8048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "d38a6d5c2145419e8f8aa1d116f69b85"
format_stamp: "Formatted at 2026-09-26 07:38:02 on dist-test-slave-qlq8"
server_key: "87358093749e93434f9c3dd0c16e2329"
server_key_iv: "f16a1cd35b172fa2c23fe1777c6c9ba4"
server_key_version: "encryptionkey@0"
I20260926 07:38:03.002509 8048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:03.029023 8048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:03.033700 8048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:03.034039 8048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:03.054836 8048 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:35335
I20260926 07:38:03.054837 8114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:35335 every 8 connection(s)
I20260926 07:38:03.056448 8048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:03.060375 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8048
I20260926 07:38:03.060451 8115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:03.060638 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:03.061396 23413 external_mini_cluster.cc:1428] Setting key ad1faab95eb4b96965b617faeb440903
I20260926 07:38:03.073064 8115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85: Bootstrap starting.
I20260926 07:38:03.076337 8115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85: Neither blocks nor log segments found. Creating new log.
Sep 26 07:38:03 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:03.077431 8115 log.cc:824] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:03.080796 8115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85: No bootstrap required, opened a new log
I20260926 07:38:03.086302 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:03.063638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.22.221.126:35335)
0926 07:38:03.064460 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:03.064471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:03.064513 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:38:03.066767 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:03.066797 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:03.067345 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:03.067828 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:03.069493 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:03.071004 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:03.072554 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:03.073355 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:03.073572 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:03.078194 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:03.078202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:03.078219 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:03.078270 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:03.081839 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:03.082621 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:03.082628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:03.082634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:03.082802 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:03.083273 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:03.083278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:03.083281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:03.083924 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:03.084266 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:03.084759 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 07:38:03.085214 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":354,"threads_started":1}
I20260926 07:38:03.089344 8115 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35335 } }
I20260926 07:38:03.089810 8115 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:03.089922 8115 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38a6d5c2145419e8f8aa1d116f69b85, State: Initialized, Role: FOLLOWER
I20260926 07:38:03.090484 8115 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35335 } }
I20260926 07:38:03.090750 8115 raft_consensus.cc:388] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:03.090915 8115 raft_consensus.cc:482] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:03.091090 8115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:03.093483 8115 raft_consensus.cc:504] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35335 } }
I20260926 07:38:03.094033 8115 leader_election.cc:302] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d38a6d5c2145419e8f8aa1d116f69b85; no voters:
I20260926 07:38:03.095100 8115 leader_election.cc:288] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:03.095217 8120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:03.095971 8120 raft_consensus.cc:686] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [term 1 LEADER]: Becoming Leader. State: Replica: d38a6d5c2145419e8f8aa1d116f69b85, State: Running, Role: LEADER
I20260926 07:38:03.096633 8120 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35335 } }
I20260926 07:38:03.097664 8115 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:03.099984 8122 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38a6d5c2145419e8f8aa1d116f69b85. Latest consensus state: current_term: 1 leader_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35335 } } }
I20260926 07:38:03.100189 8121 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38a6d5c2145419e8f8aa1d116f69b85" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35335 } } }
I20260926 07:38:03.100610 8121 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:03.100481 8122 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:03.103044 8125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:03.108513 8125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:03.119030 8125 catalog_manager.cc:1347] Generated new cluster ID: a9d31f06ebb944368a3455f33e73fe59
I20260926 07:38:03.119184 8125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:03.143994 8125 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:03.145649 8125 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:03.157347 8125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85: Generated new TSK 0
I20260926 07:38:03.158418 8125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:38:03.226577 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35335
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:03.560246 8141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:03.560525 8141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:03.560622 8141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:03.567840 8141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:03.567950 8141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:03.568063 8141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:38:03.577601 8141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:03.578819 8141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:03.580149 8141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:03.593508 8146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:03.593531 8149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:03.593900 8147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:03.594237 8141 server_base.cc:1034] running on GCE node
I20260926 07:38:03.594884 8141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:03.595902 8141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:03.597167 8141 hybrid_clock.cc:648] HybridClock initialized: now 1790408283597128 us; error 49 us; skew 500 ppm
I20260926 07:38:03.597553 8141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:03 dist-test-slave-qlq8 krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408283, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:03.606837 8141 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:38:03.608594 8141 webserver.cc:460] Webserver started at http://127.22.221.65:41911/ using document root <none> and password file <none>
I20260926 07:38:03.609319 8141 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:03.609442 8141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:03.609714 8141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:03.612888 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "8a9bd163ea294d4881fe5a2577edc011"
format_stamp: "Formatted at 2026-09-26 07:38:03 on dist-test-slave-qlq8"
server_key: "763a17b4b7f2be9d7324a3b3205f3a50"
server_key_iv: "5e37e9757c226907ce71e731b4baa153"
server_key_version: "encryptionkey@0"
I20260926 07:38:03.613657 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "8a9bd163ea294d4881fe5a2577edc011"
format_stamp: "Formatted at 2026-09-26 07:38:03 on dist-test-slave-qlq8"
server_key: "763a17b4b7f2be9d7324a3b3205f3a50"
server_key_iv: "5e37e9757c226907ce71e731b4baa153"
server_key_version: "encryptionkey@0"
I20260926 07:38:03.619112 8141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:38:03.622840 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:03.624313 8141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:03.624457 8141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "8a9bd163ea294d4881fe5a2577edc011"
format_stamp: "Formatted at 2026-09-26 07:38:03 on dist-test-slave-qlq8"
server_key: "763a17b4b7f2be9d7324a3b3205f3a50"
server_key_iv: "5e37e9757c226907ce71e731b4baa153"
server_key_version: "encryptionkey@0"
I20260926 07:38:03.624639 8141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:03.636397 8141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:03.640336 8141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:03.640579 8141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:03.641553 8141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:03.642818 8141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:03.642889 8141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:03.642968 8141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:03.643013 8141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:03.676151 8141 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:43935
I20260926 07:38:03.676237 8268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:43935 every 8 connection(s)
I20260926 07:38:03.677433 8141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:38:03.682726 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8141
I20260926 07:38:03.682935 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:38:03.683830 23413 external_mini_cluster.cc:1428] Setting key 5c103d9e9dd894b7590e89990a75107a
Sep 26 07:38:03 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408283, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:03.698696 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:03.679884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:60851 (local address 127.22.221.126:35335)
0926 07:38:03.680388 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:03.680393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:03.681250 (+ 857us) server_negotiation.cc:407] Connection header received
0926 07:38:03.683546 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:03.683550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:03.683624 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:03.683720 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:03.685568 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:03.686626 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:03.688226 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:03.688933 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:03.689076 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:03.692466 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:03.692474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:03.692482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:03.692526 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:03.695403 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:03.696318 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:03.696322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:03.696324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:03.696434 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:03.696907 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:03.696910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:03.696912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:03.697174 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:03.697293 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:03.698206 (+ 913us) server_negotiation.cc:299] Negotiation successful
0926 07:38:03.698374 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":298,"threads_started":1}
I20260926 07:38:03.700088 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:03.680385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35335 (local address 127.22.221.65:60851)
0926 07:38:03.681084 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:03.681120 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:03.683182 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:03.684001 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:03.684013 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:03.684685 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:03.685389 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:03.685406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:03.686790 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:03.686795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:03.688054 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:03.688070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:03.689156 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:03.689162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:03.689355 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:03.690043 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:03.690065 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:03.692248 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:03.695529 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:03.695541 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:03.695558 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:03.696201 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:03.696534 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:03.696542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:03.696548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:03.696795 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:03.697298 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:03.697308 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:03.697799 (+ 491us) client_negotiation.cc:769] Sending connection context
0926 07:38:03.698093 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 07:38:03.698659 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":267,"threads_started":1}
I20260926 07:38:03.701660 8269 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35335
I20260926 07:38:03.701977 8269 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:03.702709 8269 heartbeater.cc:511] Master 127.22.221.126:35335 requested a full tablet report, sending...
I20260926 07:38:03.705039 8080 ts_manager.cc:194] Registered new tserver with Master: 8a9bd163ea294d4881fe5a2577edc011 (127.22.221.65:43935)
I20260926 07:38:03.706681 8080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:60851
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:38:03.743831 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35335
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:04.079667 8275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:04.079999 8275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:04.080165 8275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:04.087973 8275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:04.088132 8275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:04.088315 8275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:38:04.098553 8275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:04.100076 8275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:04.101372 8275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:04.114650 8280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:04.114694 8283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:04.114734 8281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:04.115833 8275 server_base.cc:1034] running on GCE node
I20260926 07:38:04.116421 8275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:04.117414 8275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:04.118649 8275 hybrid_clock.cc:648] HybridClock initialized: now 1790408284118592 us; error 49 us; skew 500 ppm
I20260926 07:38:04.118975 8275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:04 dist-test-slave-qlq8 krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408284, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:04.122298 8275 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:38:04.123802 8275 webserver.cc:460] Webserver started at http://127.22.221.66:42981/ using document root <none> and password file <none>
I20260926 07:38:04.124488 8275 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:04.124634 8275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:04.124953 8275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:04.127878 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "6eca5304678446a081d97f521c50a5e2"
format_stamp: "Formatted at 2026-09-26 07:38:04 on dist-test-slave-qlq8"
server_key: "80300073d8b2f823d9ac0d65c8b5d910"
server_key_iv: "fb9435e05060ea719ec2c995064b7ab6"
server_key_version: "encryptionkey@0"
I20260926 07:38:04.128742 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "6eca5304678446a081d97f521c50a5e2"
format_stamp: "Formatted at 2026-09-26 07:38:04 on dist-test-slave-qlq8"
server_key: "80300073d8b2f823d9ac0d65c8b5d910"
server_key_iv: "fb9435e05060ea719ec2c995064b7ab6"
server_key_version: "encryptionkey@0"
I20260926 07:38:04.134116 8275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:04.137861 8290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:04.139387 8275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:04.139593 8275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "6eca5304678446a081d97f521c50a5e2"
format_stamp: "Formatted at 2026-09-26 07:38:04 on dist-test-slave-qlq8"
server_key: "80300073d8b2f823d9ac0d65c8b5d910"
server_key_iv: "fb9435e05060ea719ec2c995064b7ab6"
server_key_version: "encryptionkey@0"
I20260926 07:38:04.139809 8275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:04.165338 8275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:04.169795 8275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:04.170099 8275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:04.171129 8275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:04.172624 8275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:04.172724 8275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:04.172871 8275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:04.172945 8275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:04.206632 8275 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:40663
I20260926 07:38:04.206717 8402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:40663 every 8 connection(s)
I20260926 07:38:04.208086 8275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:38:04.209280 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8275
I20260926 07:38:04.209491 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:38:04.209890 23413 external_mini_cluster.cc:1428] Setting key aa1a2a59f298d209f386274fe29ff33a
Sep 26 07:38:04 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408284, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:04.234117 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.212231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:40771 (local address 127.22.221.126:35335)
0926 07:38:04.212669 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.212675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.213879 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 07:38:04.216730 (+ 2851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.216734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.216801 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.216898 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:04.219107 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.219886 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.222135 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.222874 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.223018 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.226712 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:04.226720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:04.226728 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:04.226774 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:04.230503 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:04.231445 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:04.231450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:04.231477 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:04.231636 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:04.232155 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:04.232160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:04.232164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:04.232515 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:04.232652 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.233636 (+ 984us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.233823 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
I20260926 07:38:04.235745 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.212877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35335 (local address 127.22.221.66:40771)
0926 07:38:04.213708 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.213759 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.216395 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.217186 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.217200 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.218039 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:04.218881 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.218909 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.220050 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.220058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.221961 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.221973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.223112 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.223117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.223386 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.224108 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:04.224130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:04.226439 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:04.230653 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:04.230660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:04.230673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:04.231301 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:04.231753 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:04.231761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:04.231767 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:04.232025 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:04.232673 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:04.232684 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:04.233203 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.233519 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.234159 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":294,"threads_started":1}
I20260926 07:38:04.237362 8403 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35335
I20260926 07:38:04.237684 8403 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:04.238461 8403 heartbeater.cc:511] Master 127.22.221.126:35335 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
I20260926 07:38:04.240141 8080 ts_manager.cc:194] Registered new tserver with Master: 6eca5304678446a081d97f521c50a5e2 (127.22.221.66:40663)
I20260926 07:38:04.240996 8080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:40771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:38:04.272480 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35335
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:04.604310 8409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:04.604578 8409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:04.604715 8409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:04.612198 8409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:04.612308 8409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:04.612421 8409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:38:04.622042 8409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:04.623327 8409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:04.624594 8409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:04.637661 8415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:04.637741 8417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:04.638255 8414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:04.638608 8409 server_base.cc:1034] running on GCE node
I20260926 07:38:04.639276 8409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:04.640208 8409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:04.641450 8409 hybrid_clock.cc:648] HybridClock initialized: now 1790408284641392 us; error 48 us; skew 500 ppm
I20260926 07:38:04.641788 8409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:04 dist-test-slave-qlq8 krb5kdc[8035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408284, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:04.645120 8409 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:38:04.646564 8409 webserver.cc:460] Webserver started at http://127.22.221.67:46543/ using document root <none> and password file <none>
I20260926 07:38:04.647296 8409 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:04.647441 8409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:04.647753 8409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:04.650552 8409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "e6eebfbeb929447583b2fc60eeb5d919"
format_stamp: "Formatted at 2026-09-26 07:38:04 on dist-test-slave-qlq8"
server_key: "0b2320cb282e2783212fe48bd28c18ab"
server_key_iv: "727d11c0c1f31323e928b66201d9fdb7"
server_key_version: "encryptionkey@0"
I20260926 07:38:04.651422 8409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "e6eebfbeb929447583b2fc60eeb5d919"
format_stamp: "Formatted at 2026-09-26 07:38:04 on dist-test-slave-qlq8"
server_key: "0b2320cb282e2783212fe48bd28c18ab"
server_key_iv: "727d11c0c1f31323e928b66201d9fdb7"
server_key_version: "encryptionkey@0"
I20260926 07:38:04.656461 8409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:38:04.659716 8424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:04.660962 8409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:38:04.661139 8409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "e6eebfbeb929447583b2fc60eeb5d919"
format_stamp: "Formatted at 2026-09-26 07:38:04 on dist-test-slave-qlq8"
server_key: "0b2320cb282e2783212fe48bd28c18ab"
server_key_iv: "727d11c0c1f31323e928b66201d9fdb7"
server_key_version: "encryptionkey@0"
I20260926 07:38:04.661321 8409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:04.702370 8409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:04.706954 8409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:04.707280 8409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:04.708338 8409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:04.709817 8409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:04.709941 8409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:04.710089 8409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:04.710158 8409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:04.710762 8269 heartbeater.cc:503] Master 127.22.221.126:35335 was elected leader, sending a full tablet report...
I20260926 07:38:04.745318 8409 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:42603
I20260926 07:38:04.745368 8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:42603 every 8 connection(s)
I20260926 07:38:04.746685 8409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:38:04.748422 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8409
I20260926 07:38:04.748621 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:38:04.749212 23413 external_mini_cluster.cc:1428] Setting key 21090ae102040da90b05cea1f8a63281
Sep 26 07:38:04 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408284, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:04.767787 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.749624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:44715 (local address 127.22.221.126:35335)
0926 07:38:04.750156 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.750163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.751415 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 07:38:04.753567 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.753571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.753648 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.753743 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:04.755363 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.756278 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.757823 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.758216 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.758318 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.761714 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:04.761719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:04.761723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:04.761757 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:04.764807 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:04.765643 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:04.765647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:04.765650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:04.765813 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:04.766231 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:04.766234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:04.766236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:04.766496 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:04.766616 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.767342 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.767512 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":249,"threads_started":1}
I20260926 07:38:04.768761 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.750266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35335 (local address 127.22.221.67:44715)
0926 07:38:04.751184 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.751278 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.753261 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.753958 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.753966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.754520 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:04.755121 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.755139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.756418 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.756421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.757678 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.757689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.758410 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.758414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.758634 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.759364 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:04.759386 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:04.761515 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:04.764970 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:04.764976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:04.764988 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:04.765540 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:04.765925 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:04.765928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:04.765931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:04.766136 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:04.766615 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:04.766621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:04.766944 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.767198 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.767671 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":374,"threads_started":1}
I20260926 07:38:04.770184 8537 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35335
I20260926 07:38:04.770503 8537 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:04.771347 8537 heartbeater.cc:511] Master 127.22.221.126:35335 requested a full tablet report, sending...
I20260926 07:38:04.772756 8080 ts_manager.cc:194] Registered new tserver with Master: e6eebfbeb929447583b2fc60eeb5d919 (127.22.221.67:42603)
I20260926 07:38:04.773725 8080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:44715
I20260926 07:38:04.777874 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:38:04.796367 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.782703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.22.221.126:35335)
0926 07:38:04.782930 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.782940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.783390 (+ 450us) server_negotiation.cc:407] Connection header received
0926 07:38:04.783589 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.783597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.783664 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.783756 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:04.785638 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.786392 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.788052 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.788489 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.788607 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.791200 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:04.791205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:04.791208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:04.791307 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:04.793945 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:04.794589 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:04.794593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:04.794595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:04.794709 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:04.795024 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:04.795027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:04.795030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:04.795399 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:04.795598 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.795940 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.796102 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:38:04.800441 8080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:38:04.807520 8080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:38:04.832122 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.820132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.22.221.66:40663)
0926 07:38:04.821823 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.821831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.821857 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:38:04.821958 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.821965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.822225 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.823363 (+ 1138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:04.823676 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.825390 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.829037 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.830593 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.830732 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.830956 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.831628 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.831809 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1535,"thread_start_us":1431,"threads_started":1}
I20260926 07:38:04.832494 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.819910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:40663 (local address 127.0.0.1:49818)
0926 07:38:04.820943 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.820995 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.821212 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.822458 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.822466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.822497 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:04.823070 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.823086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.825592 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.825602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.828777 (+ 3175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.828800 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.830797 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.830804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.831043 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.831101 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.831317 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.831520 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":1793,"threads_started":1}
I20260926 07:38:04.836642 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.820679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.22.221.67:42603)
0926 07:38:04.829392 (+ 8713us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.829401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.829433 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:38:04.829530 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.829538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.829789 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.829993 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:04.830778 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.832405 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.834533 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.835564 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.835713 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.835941 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.836164 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.836311 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8510,"thread_start_us":8351,"threads_started":1}
I20260926 07:38:04.836660 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.820150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:42603 (local address 127.0.0.1:59234)
0926 07:38:04.821719 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.821736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.821850 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.830006 (+ 8156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.830014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.830048 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:04.830516 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.830527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.832585 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.832593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.834365 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.834379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.835774 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.835781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.835967 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.836007 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.836113 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.836249 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"thread_start_us":538,"threads_started":1}
I20260926 07:38:04.837100 8338 tablet_service.cc:1467] Processing CreateTablet for tablet ba34baf837534bb08e15cb07140ff0f7 (DEFAULT_TABLE table=test-table [id=80a3337b98eb44fdb347952be7de9977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:04.838625 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba34baf837534bb08e15cb07140ff0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:04.839421 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.822048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:43935 (local address 127.0.0.1:49386)
0926 07:38:04.829775 (+ 7727us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.829790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.829877 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.832203 (+ 2326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.832211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.832239 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:04.832942 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.832955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.834595 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.834602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.836644 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.836662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.838558 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.838565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.838755 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.838793 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.838885 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.839027 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7614,"mutex_wait_us":1,"thread_start_us":7487,"threads_started":1}
I20260926 07:38:04.839717 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.822380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.22.221.65:43935)
0926 07:38:04.830724 (+ 8344us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.830733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.830759 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:38:04.830850 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.830878 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.831106 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.831337 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:04.833166 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.834468 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.836838 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.838247 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.838382 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.838616 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.838953 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.839116 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8152,"thread_start_us":7994,"threads_started":1}
I20260926 07:38:04.840561 8472 tablet_service.cc:1467] Processing CreateTablet for tablet ba34baf837534bb08e15cb07140ff0f7 (DEFAULT_TABLE table=test-table [id=80a3337b98eb44fdb347952be7de9977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:04.841854 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba34baf837534bb08e15cb07140ff0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:04.843580 8204 tablet_service.cc:1467] Processing CreateTablet for tablet ba34baf837534bb08e15cb07140ff0f7 (DEFAULT_TABLE table=test-table [id=80a3337b98eb44fdb347952be7de9977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:04.844974 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba34baf837534bb08e15cb07140ff0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:04.852929 8556 tablet_bootstrap.cc:492] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Bootstrap starting.
I20260926 07:38:04.854941 8557 tablet_bootstrap.cc:492] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Bootstrap starting.
I20260926 07:38:04.856840 8556 tablet_bootstrap.cc:654] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:04.858130 8556 log.cc:824] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:04.858414 8557 tablet_bootstrap.cc:654] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:04.858825 8558 tablet_bootstrap.cc:492] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Bootstrap starting.
I20260926 07:38:04.859839 8557 log.cc:824] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:04.862027 8558 tablet_bootstrap.cc:654] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:04.862954 8558 log.cc:824] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:04.863267 8556 tablet_bootstrap.cc:492] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: No bootstrap required, opened a new log
I20260926 07:38:04.863596 8556 ts_tablet_manager.cc:1397] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 07:38:04.865964 8558 tablet_bootstrap.cc:492] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: No bootstrap required, opened a new log
I20260926 07:38:04.865967 8557 tablet_bootstrap.cc:492] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: No bootstrap required, opened a new log
I20260926 07:38:04.866233 8557 ts_tablet_manager.cc:1397] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 07:38:04.866233 8558 ts_tablet_manager.cc:1397] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 07:38:04.871289 8556 raft_consensus.cc:348] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.871744 8556 raft_consensus.cc:374] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:04.871851 8556 raft_consensus.cc:729] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6eebfbeb929447583b2fc60eeb5d919, State: Initialized, Role: FOLLOWER
I20260926 07:38:04.872447 8556 consensus_queue.cc:260] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.873492 8537 heartbeater.cc:503] Master 127.22.221.126:35335 was elected leader, sending a full tablet report...
I20260926 07:38:04.875434 8556 ts_tablet_manager.cc:1428] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260926 07:38:04.876977 8558 raft_consensus.cc:348] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.876989 8557 raft_consensus.cc:348] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.877429 8557 raft_consensus.cc:374] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:04.877432 8558 raft_consensus.cc:374] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:04.877537 8557 raft_consensus.cc:729] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eca5304678446a081d97f521c50a5e2, State: Initialized, Role: FOLLOWER
I20260926 07:38:04.877537 8558 raft_consensus.cc:729] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9bd163ea294d4881fe5a2577edc011, State: Initialized, Role: FOLLOWER
I20260926 07:38:04.878162 8557 consensus_queue.cc:260] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.878163 8558 consensus_queue.cc:260] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.879441 8403 heartbeater.cc:503] Master 127.22.221.126:35335 was elected leader, sending a full tablet report...
I20260926 07:38:04.880249 8557 ts_tablet_manager.cc:1428] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260926 07:38:04.880447 8558 ts_tablet_manager.cc:1428] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 07:38:04.889427 8563 raft_consensus.cc:482] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:04.889673 8563 raft_consensus.cc:504] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.891340 8563 leader_election.cc:288] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6eebfbeb929447583b2fc60eeb5d919 (127.22.221.67:42603), 8a9bd163ea294d4881fe5a2577edc011 (127.22.221.65:43935)
I20260926 07:38:04.898227 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.891781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:52713 (local address 127.22.221.67:42603)
0926 07:38:04.892013 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.892021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.892806 (+ 785us) server_negotiation.cc:407] Connection header received
0926 07:38:04.893052 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.893056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.893116 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.893208 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:04.893758 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.894481 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.896416 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.897202 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.897417 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.897495 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.897826 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.897954 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:38:04.898281 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.892054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:42217 (local address 127.22.221.65:43935)
0926 07:38:04.892307 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.892315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.893060 (+ 745us) server_negotiation.cc:407] Connection header received
0926 07:38:04.893262 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.893272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.893330 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.893426 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:04.893977 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.894685 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.896494 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.897274 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.897479 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.897557 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.897889 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.898031 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:38:04.898370 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.891701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:42603 (local address 127.22.221.66:52713)
0926 07:38:04.892685 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.892705 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.892890 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.893225 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.893229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.893261 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:04.893619 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.893628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.894612 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.894618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.896259 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.896272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.897440 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.897447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.897606 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.897634 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.897800 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.897938 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":21,"thread_start_us":911,"threads_started":1}
I20260926 07:38:04.898830 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.891787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:43935 (local address 127.22.221.66:42217)
0926 07:38:04.892942 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:04.892958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:04.893111 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:04.893449 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:04.893454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:04.893478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:04.893831 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.893839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.894825 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.894831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.896332 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:04.896344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.897529 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.897535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:04.897740 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.897764 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:38:04.897863 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:38:04.897998 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":4,"thread_start_us":437,"threads_started":1}
I20260926 07:38:04.899443 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba34baf837534bb08e15cb07140ff0f7" candidate_uuid: "6eca5304678446a081d97f521c50a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6eebfbeb929447583b2fc60eeb5d919" is_pre_election: true
I20260926 07:38:04.900018 8492 raft_consensus.cc:2454] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eca5304678446a081d97f521c50a5e2 in term 0.
I20260926 07:38:04.899950 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba34baf837534bb08e15cb07140ff0f7" candidate_uuid: "6eca5304678446a081d97f521c50a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9bd163ea294d4881fe5a2577edc011" is_pre_election: true
I20260926 07:38:04.900532 8224 raft_consensus.cc:2454] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eca5304678446a081d97f521c50a5e2 in term 0.
I20260926 07:38:04.900849 8291 leader_election.cc:302] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eca5304678446a081d97f521c50a5e2, e6eebfbeb929447583b2fc60eeb5d919; no voters:
I20260926 07:38:04.901315 8563 raft_consensus.cc:2781] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:38:04.901477 8563 raft_consensus.cc:482] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:38:04.901587 8563 raft_consensus.cc:3037] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:04.903860 8563 raft_consensus.cc:504] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.904469 8563 leader_election.cc:288] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [CANDIDATE]: Term 1 election: Requested vote from peers e6eebfbeb929447583b2fc60eeb5d919 (127.22.221.67:42603), 8a9bd163ea294d4881fe5a2577edc011 (127.22.221.65:43935)
I20260926 07:38:04.904865 8492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba34baf837534bb08e15cb07140ff0f7" candidate_uuid: "6eca5304678446a081d97f521c50a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6eebfbeb929447583b2fc60eeb5d919"
I20260926 07:38:04.904925 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba34baf837534bb08e15cb07140ff0f7" candidate_uuid: "6eca5304678446a081d97f521c50a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9bd163ea294d4881fe5a2577edc011"
I20260926 07:38:04.905077 8492 raft_consensus.cc:3037] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:04.905185 8224 raft_consensus.cc:3037] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:04.907702 8224 raft_consensus.cc:2454] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eca5304678446a081d97f521c50a5e2 in term 1.
I20260926 07:38:04.907732 8492 raft_consensus.cc:2454] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eca5304678446a081d97f521c50a5e2 in term 1.
I20260926 07:38:04.908187 8292 leader_election.cc:302] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eca5304678446a081d97f521c50a5e2, 8a9bd163ea294d4881fe5a2577edc011; no voters:
I20260926 07:38:04.908517 8563 raft_consensus.cc:2781] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:04.908991 8563 raft_consensus.cc:686] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 1 LEADER]: Becoming Leader. State: Replica: 6eca5304678446a081d97f521c50a5e2, State: Running, Role: LEADER
I20260926 07:38:04.909505 8563 consensus_queue.cc:237] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } }
I20260926 07:38:04.914371 8080 catalog_manager.cc:5495] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eca5304678446a081d97f521c50a5e2 (127.22.221.66). New cstate: current_term: 1 leader_uuid: "6eca5304678446a081d97f521c50a5e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eca5304678446a081d97f521c50a5e2" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 } health_report { overall_health: UNKNOWN } } }
W20260926 07:38:04.929759 8270 tablet.cc:2369] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:04.932554 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.926860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.22.221.66:40663)
0926 07:38:04.927080 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.927086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.927214 (+ 128us) server_negotiation.cc:407] Connection header received
0926 07:38:04.927487 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.927491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.927592 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.927747 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:38:04.928110 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.928830 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.930349 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.930763 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.930913 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:04.931441 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:38:04.931931 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:38:04.932094 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:04.932193 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:38:04.932274 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:38:04.940958 8224 raft_consensus.cc:1261] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 1 FOLLOWER]: Refusing update from remote peer 6eca5304678446a081d97f521c50a5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:04.940965 8492 raft_consensus.cc:1261] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 1 FOLLOWER]: Refusing update from remote peer 6eca5304678446a081d97f521c50a5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:04.941924 8567 consensus_queue.cc:1035] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9bd163ea294d4881fe5a2577edc011" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:04.942363 8563 consensus_queue.cc:1035] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6eebfbeb929447583b2fc60eeb5d919" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:04.954532 8570 mvcc.cc:205] Tried to move back new op lower bound from 7333512335107461120 to 7333512334993522688. Current Snapshot: MvccSnapshot[applied={T|T < 7333512335107461120}]
I20260926 07:38:04.969926 8079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:04.970291 8079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:04.974509 8079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38a6d5c2145419e8f8aa1d116f69b85: Sending DeleteTablet for 3 replicas of tablet ba34baf837534bb08e15cb07140ff0f7
I20260926 07:38:04.976156 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet ba34baf837534bb08e15cb07140ff0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:04 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49386
I20260926 07:38:04.976352 8472 tablet_service.cc:1514] Processing DeleteTablet for tablet ba34baf837534bb08e15cb07140ff0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:04 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:59234
I20260926 07:38:04.977092 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet ba34baf837534bb08e15cb07140ff0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:04 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:49818
I20260926 07:38:04.977270 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8141
I20260926 07:38:04.988683 8580 tablet_replica.cc:330] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: stopping tablet replica
I20260926 07:38:04.989409 8580 raft_consensus.cc:2229] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:04.990154 8580 raft_consensus.cc:2258] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:04.992885 8580 ts_tablet_manager.cc:1905] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:04.997520 8581 tablet_replica.cc:330] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: stopping tablet replica
I20260926 07:38:04.998154 8581 raft_consensus.cc:2229] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:04.998821 8581 raft_consensus.cc:2258] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:04.999769 8582 tablet_replica.cc:330] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: stopping tablet replica
I20260926 07:38:05.000188 8582 raft_consensus.cc:2229] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:38:05.000912 8582 raft_consensus.cc:2258] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:05.001672 8580 ts_tablet_manager.cc:1918] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:05.001927 8580 log.cc:1196] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/ba34baf837534bb08e15cb07140ff0f7
I20260926 07:38:05.002614 8580 ts_tablet_manager.cc:1939] T ba34baf837534bb08e15cb07140ff0f7 P 8a9bd163ea294d4881fe5a2577edc011: Deleting consensus metadata
I20260926 07:38:05.003706 8582 ts_tablet_manager.cc:1905] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:05.004871 8065 catalog_manager.cc:4841] TS 8a9bd163ea294d4881fe5a2577edc011 (127.22.221.65:43935): tablet ba34baf837534bb08e15cb07140ff0f7 (table test-table [id=80a3337b98eb44fdb347952be7de9977]) successfully deleted
Sep 26 07:38:05 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:38:05.007491 8581 ts_tablet_manager.cc:1905] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:05.012073 8582 ts_tablet_manager.cc:1918] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:05.012254 8582 log.cc:1196] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/ba34baf837534bb08e15cb07140ff0f7
I20260926 07:38:05.012821 8582 ts_tablet_manager.cc:1939] T ba34baf837534bb08e15cb07140ff0f7 P 6eca5304678446a081d97f521c50a5e2: Deleting consensus metadata
I20260926 07:38:05.014285 8581 ts_tablet_manager.cc:1918] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:05.014487 8581 log.cc:1196] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/ba34baf837534bb08e15cb07140ff0f7
I20260926 07:38:05.014597 8065 catalog_manager.cc:4841] TS 6eca5304678446a081d97f521c50a5e2 (127.22.221.66:40663): tablet ba34baf837534bb08e15cb07140ff0f7 (table test-table [id=80a3337b98eb44fdb347952be7de9977]) successfully deleted
I20260926 07:38:05.015020 8581 ts_tablet_manager.cc:1939] T ba34baf837534bb08e15cb07140ff0f7 P e6eebfbeb929447583b2fc60eeb5d919: Deleting consensus metadata
I20260926 07:38:05.015180 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:04.978880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.22.221.65:43935)
0926 07:38:04.979114 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:04.979123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:04.989256 (+ 10133us) server_negotiation.cc:407] Connection header received
0926 07:38:04.989776 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:04.989785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:04.989927 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:04.990036 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:04.993119 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:04.994524 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:04.999641 (+ 5117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:05.000293 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:05.000418 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:05.006269 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:05.006286 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:05.006307 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:05.006380 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:05.010881 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:05.011925 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:05.011934 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:05.011939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:05.012125 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:05.012786 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:05.012791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:05.012795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:05.013344 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:05.013677 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:05.014312 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 07:38:05.014711 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:38:05.016690 8064 catalog_manager.cc:4841] TS e6eebfbeb929447583b2fc60eeb5d919 (127.22.221.67:42603): tablet ba34baf837534bb08e15cb07140ff0f7 (table test-table [id=80a3337b98eb44fdb347952be7de9977]) successfully deleted
I20260926 07:38:05.016745 8264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:05.107991 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8141
I20260926 07:38:05.125790 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8275
Sep 26 07:38:05 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:38:05.142623 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:05.126681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.22.221.66:40663)
0926 07:38:05.126905 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:05.126912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:05.126936 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:38:05.127066 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:05.127072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:05.127160 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:05.127332 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:05.128643 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:05.130072 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:05.131450 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:05.131912 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:05.132047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:05.136352 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:05.136365 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:05.136381 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:05.136424 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:05.139708 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:05.140528 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:05.140535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:05.140537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:05.140675 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:05.141061 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:05.141068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:05.141075 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:05.141433 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:05.141783 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:05.142051 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 07:38:05.142277 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 07:38:05.143469 8398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:05.254956 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8275
I20260926 07:38:05.275688 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8409
Sep 26 07:38:05 dist-test-slave-qlq8 krb5kdc[8035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:38:05.291126 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:05.276459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.22.221.67:42603)
0926 07:38:05.276664 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:05.276672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:05.276804 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:38:05.277081 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:05.277085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:05.277149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:05.277252 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:05.278567 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:05.279375 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:05.280742 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:05.281157 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:05.281295 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:05.285737 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:05.285751 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:05.285768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:05.285804 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:05.288485 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:05.289289 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:05.289295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:05.289298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:05.289444 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:05.289769 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:05.289772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:05.289774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:05.290134 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:05.290343 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:05.290709 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:38:05.290888 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:38:05.291817 8532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:05.381971 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8409
I20260926 07:38:05.398471 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8048
I20260926 07:38:05.399729 8110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:05.522079 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8048
2026-09-26T07:38:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4280 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:05Z Disabled control of system clock
I20260926 07:38:05.600136 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35945
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:38:05.929000 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:05.929297 8591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:05.929441 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:05.936939 8591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:05.937057 8591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:05.937132 8591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:05.937220 8591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:05.937302 8591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:05.947039 8591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:05.948396 8591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:05.950213 8591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:05.962101 8596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:05.962119 8597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:05.962055 8599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:05.963294 8591 server_base.cc:1034] running on GCE node
I20260926 07:38:05.964075 8591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:05.965435 8591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:05.966638 8591 hybrid_clock.cc:648] HybridClock initialized: now 1790408285966572 us; error 55 us; skew 500 ppm
I20260926 07:38:05.967017 8591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:05.969350 8591 webserver.cc:460] Webserver started at http://127.22.221.126:42025/ using document root <none> and password file <none>
I20260926 07:38:05.970073 8591 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:05.970191 8591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:05.970508 8591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:05.973634 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5"
format_stamp: "Formatted at 2026-09-26 07:38:05 on dist-test-slave-qlq8"
server_key: "93f0ae030b40ac74bcfd4c9b5470af8f"
server_key_iv: "f192443bd9f6e28be045a1d0a0be36e7"
server_key_version: "encryptionkey@0"
I20260926 07:38:05.974529 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5"
format_stamp: "Formatted at 2026-09-26 07:38:05 on dist-test-slave-qlq8"
server_key: "93f0ae030b40ac74bcfd4c9b5470af8f"
server_key_iv: "f192443bd9f6e28be045a1d0a0be36e7"
server_key_version: "encryptionkey@0"
I20260926 07:38:05.979910 8591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:38:05.983561 8605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:05.984941 8591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:05.985174 8591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5"
format_stamp: "Formatted at 2026-09-26 07:38:05 on dist-test-slave-qlq8"
server_key: "93f0ae030b40ac74bcfd4c9b5470af8f"
server_key_iv: "f192443bd9f6e28be045a1d0a0be36e7"
server_key_version: "encryptionkey@0"
I20260926 07:38:05.985395 8591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:06.004086 8591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:06.005002 8591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:06.005313 8591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:06.024277 8591 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:35945
I20260926 07:38:06.024327 8656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:35945 every 8 connection(s)
I20260926 07:38:06.025658 8591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:06.029515 8657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:06.033010 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8591
I20260926 07:38:06.033221 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:06.033689 23413 external_mini_cluster.cc:1428] Setting key b9da8429216a865e96d766b17e5a85a5
I20260926 07:38:06.041921 8657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5: Bootstrap starting.
I20260926 07:38:06.045591 8657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:06.046841 8657 log.cc:824] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:06.047703 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:06.035567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.22.221.126:35945)
0926 07:38:06.036593 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:06.036604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:06.036646 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:38:06.038306 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:06.038328 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:06.038815 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:06.039226 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:06.039832 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:06.041077 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:06.042618 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:06.043425 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:06.043649 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:06.044938 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:06.044946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:06.044960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:06.045552 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:06.045646 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:06.045668 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:06.045805 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:06.046136 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:38:06.046514 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":326,"threads_started":1}
I20260926 07:38:06.050365 8657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5: No bootstrap required, opened a new log
I20260926 07:38:06.059170 8657 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35945 } }
I20260926 07:38:06.059844 8657 raft_consensus.cc:374] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:06.059967 8657 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61bb1685c4e42d1ab608b5bbbd45ab5, State: Initialized, Role: FOLLOWER
I20260926 07:38:06.060577 8657 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35945 } }
I20260926 07:38:06.060813 8657 raft_consensus.cc:388] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:06.060956 8657 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:06.061117 8657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:06.064041 8657 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35945 } }
I20260926 07:38:06.064591 8657 leader_election.cc:302] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e61bb1685c4e42d1ab608b5bbbd45ab5; no voters:
I20260926 07:38:06.065480 8657 leader_election.cc:288] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:06.065732 8662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:06.066828 8662 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [term 1 LEADER]: Becoming Leader. State: Replica: e61bb1685c4e42d1ab608b5bbbd45ab5, State: Running, Role: LEADER
I20260926 07:38:06.067462 8662 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35945 } }
I20260926 07:38:06.067724 8657 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:06.070474 8664 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61bb1685c4e42d1ab608b5bbbd45ab5. Latest consensus state: current_term: 1 leader_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35945 } } }
I20260926 07:38:06.070549 8663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61bb1685c4e42d1ab608b5bbbd45ab5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35945 } } }
I20260926 07:38:06.070974 8663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:06.070971 8664 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:06.071914 8671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:06.077378 8671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:06.086223 8671 catalog_manager.cc:1347] Generated new cluster ID: a37558e9f0f34aacb0fa2dc39b5247da
I20260926 07:38:06.086351 8671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:06.146130 8671 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:06.147670 8671 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:06.156560 8671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5: Generated new TSK 0
I20260926 07:38:06.157588 8671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:06.168855 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35945
--builtin_ntp_servers=127.22.221.84:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:38:06.505214 8681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:06.505504 8681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:06.505656 8681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:06.513012 8681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:06.513113 8681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:06.513321 8681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:38:06.522413 8681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:06.523710 8681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:06.525619 8681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:06.539135 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:06.539182 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:06.539090 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:06.539971 8681 server_base.cc:1034] running on GCE node
I20260926 07:38:06.540612 8681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:06.541563 8681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:06.542773 8681 hybrid_clock.cc:648] HybridClock initialized: now 1790408286542732 us; error 40 us; skew 500 ppm
I20260926 07:38:06.543104 8681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:06.545457 8681 webserver.cc:460] Webserver started at http://127.22.221.65:43469/ using document root <none> and password file <none>
I20260926 07:38:06.546137 8681 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:06.546224 8681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:06.546546 8681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:06.549647 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "0eeef545cc684d98a8bde4d9e7f63d6b"
format_stamp: "Formatted at 2026-09-26 07:38:06 on dist-test-slave-qlq8"
server_key: "d68a6450597e6998592f0655b2fa9008"
server_key_iv: "c191809dcdfecd315cb33a539463ae60"
server_key_version: "encryptionkey@0"
I20260926 07:38:06.550441 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "0eeef545cc684d98a8bde4d9e7f63d6b"
format_stamp: "Formatted at 2026-09-26 07:38:06 on dist-test-slave-qlq8"
server_key: "d68a6450597e6998592f0655b2fa9008"
server_key_iv: "c191809dcdfecd315cb33a539463ae60"
server_key_version: "encryptionkey@0"
I20260926 07:38:06.555505 8681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:38:06.558789 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:06.560115 8681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:06.560287 8681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "0eeef545cc684d98a8bde4d9e7f63d6b"
format_stamp: "Formatted at 2026-09-26 07:38:06 on dist-test-slave-qlq8"
server_key: "d68a6450597e6998592f0655b2fa9008"
server_key_iv: "c191809dcdfecd315cb33a539463ae60"
server_key_version: "encryptionkey@0"
I20260926 07:38:06.560499 8681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:06.578702 8681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:06.579581 8681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:06.579882 8681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:06.580833 8681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:06.582237 8681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:06.582358 8681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:06.582495 8681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:06.582582 8681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:06.616526 8681 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:45407
I20260926 07:38:06.616734 8807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:45407 every 8 connection(s)
I20260926 07:38:06.617939 8681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:38:06.627259 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8681
I20260926 07:38:06.627444 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:38:06.627974 23413 external_mini_cluster.cc:1428] Setting key fca04e7a735443b273052c7f98d0ba22
I20260926 07:38:06.631152 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35945
--builtin_ntp_servers=127.22.221.84:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:38:06.631687 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:06.620637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:58893 (local address 127.22.221.126:35945)
0926 07:38:06.621108 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:06.621112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:06.622308 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 07:38:06.624210 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:06.624217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:06.624305 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:06.624437 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:06.625512 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:06.626259 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:06.628456 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:06.629052 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:06.629218 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:06.630644 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:06.630650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:06.630655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:06.630677 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:06.630739 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:06.630760 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:06.630884 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:06.631142 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 07:38:06.631268 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":217,"threads_started":1}
I20260926 07:38:06.632643 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:06.621246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35945 (local address 127.22.221.65:58893)
0926 07:38:06.622148 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:06.622183 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:06.623938 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:06.624681 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:06.624690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:06.624783 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:06.625344 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:06.625359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:06.626404 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:06.626412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:06.628308 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:06.628323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:06.629306 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:06.629316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:06.629564 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:06.630319 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:06.630322 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:06.630369 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:06.630460 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:06.630906 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:06.630915 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:06.630921 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:38:06.631094 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:38:06.631548 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260926 07:38:06.634204 8808 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35945
I20260926 07:38:06.634511 8808 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:06.635293 8808 heartbeater.cc:511] Master 127.22.221.126:35945 requested a full tablet report, sending...
I20260926 07:38:06.637296 8622 ts_manager.cc:194] Registered new tserver with Master: 0eeef545cc684d98a8bde4d9e7f63d6b (127.22.221.65:45407)
I20260926 07:38:06.638775 8622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.65:58893
W20260926 07:38:06.962522 8812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:06.962824 8812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:06.962980 8812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:06.970685 8812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:06.970786 8812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:06.971014 8812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:38:06.980456 8812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:06.981721 8812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:06.983560 8812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:06.996959 8817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:06.996917 8820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:06.996944 8818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:06.997722 8812 server_base.cc:1034] running on GCE node
I20260926 07:38:06.998404 8812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:06.999380 8812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:07.000623 8812 hybrid_clock.cc:648] HybridClock initialized: now 1790408287000588 us; error 58 us; skew 500 ppm
I20260926 07:38:07.000954 8812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:07.003317 8812 webserver.cc:460] Webserver started at http://127.22.221.66:39975/ using document root <none> and password file <none>
I20260926 07:38:07.004001 8812 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:07.004132 8812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:07.004449 8812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:07.007418 8812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "9172d8810cc648eda58e6d435b37882c"
format_stamp: "Formatted at 2026-09-26 07:38:07 on dist-test-slave-qlq8"
server_key: "37e15af426c0bfe8a05892c4b5d1dc98"
server_key_iv: "5c7dd73fee35bc1af6beff1cce3992dc"
server_key_version: "encryptionkey@0"
I20260926 07:38:07.008200 8812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "9172d8810cc648eda58e6d435b37882c"
format_stamp: "Formatted at 2026-09-26 07:38:07 on dist-test-slave-qlq8"
server_key: "37e15af426c0bfe8a05892c4b5d1dc98"
server_key_iv: "5c7dd73fee35bc1af6beff1cce3992dc"
server_key_version: "encryptionkey@0"
I20260926 07:38:07.013141 8812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:38:07.016489 8826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:07.017752 8812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:07.017926 8812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "9172d8810cc648eda58e6d435b37882c"
format_stamp: "Formatted at 2026-09-26 07:38:07 on dist-test-slave-qlq8"
server_key: "37e15af426c0bfe8a05892c4b5d1dc98"
server_key_iv: "5c7dd73fee35bc1af6beff1cce3992dc"
server_key_version: "encryptionkey@0"
I20260926 07:38:07.018131 8812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:07.035287 8812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:07.036093 8812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:07.036402 8812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:07.037504 8812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:07.038955 8812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:07.039076 8812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:07.039211 8812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:07.039331 8812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 07:38:07.073206 8812 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:43147
I20260926 07:38:07.073266 8938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:43147 every 8 connection(s)
I20260926 07:38:07.074566 8812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:38:07.077870 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8812
I20260926 07:38:07.078066 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:38:07.078541 23413 external_mini_cluster.cc:1428] Setting key 1dcb70de0cea95c28a72b8ee9ffbf6b2
I20260926 07:38:07.081473 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35945
--builtin_ntp_servers=127.22.221.84:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:38:07.087713 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.077270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:48189 (local address 127.22.221.126:35945)
0926 07:38:07.077446 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.077451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.079103 (+ 1652us) server_negotiation.cc:407] Connection header received
0926 07:38:07.081212 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.081218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.081298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.081427 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:07.082584 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.083379 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.084932 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.085340 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.085481 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.086875 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:07.086881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:07.086886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:07.086915 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:07.086947 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:07.086959 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:07.087056 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.087293 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.087408 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:38:07.088742 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.077828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35945 (local address 127.22.221.66:48189)
0926 07:38:07.078920 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.078976 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.080954 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.081700 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.081713 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.081816 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:07.082424 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.082442 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.083528 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.083538 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.084772 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.084783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.085581 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.085588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.085821 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.086565 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:07.086567 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:07.086606 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:07.086686 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:07.087054 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:07.087067 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:07.087077 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.087293 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.087689 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":369,"threads_started":1}
I20260926 07:38:07.090221 8939 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35945
I20260926 07:38:07.090498 8939 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:07.091210 8939 heartbeater.cc:511] Master 127.22.221.126:35945 requested a full tablet report, sending...
I20260926 07:38:07.092804 8622 ts_manager.cc:194] Registered new tserver with Master: 9172d8810cc648eda58e6d435b37882c (127.22.221.66:43147)
I20260926 07:38:07.093683 8622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.66:48189
W20260926 07:38:07.407585 8942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:07.407893 8942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:07.408075 8942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:07.415612 8942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:07.415740 8942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:07.415957 8942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:38:07.425031 8942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:35945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:07.426291 8942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:07.428148 8942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:07.441207 8948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:07.441263 8950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:07.441264 8947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:07.442157 8942 server_base.cc:1034] running on GCE node
I20260926 07:38:07.442739 8942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:07.443720 8942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:07.444960 8942 hybrid_clock.cc:648] HybridClock initialized: now 1790408287444886 us; error 24 us; skew 500 ppm
I20260926 07:38:07.445320 8942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:07.447546 8942 webserver.cc:460] Webserver started at http://127.22.221.67:41545/ using document root <none> and password file <none>
I20260926 07:38:07.448244 8942 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:07.448354 8942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:07.448676 8942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:07.451673 8942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "c106df996c6b46c58e77e91916611eca"
format_stamp: "Formatted at 2026-09-26 07:38:07 on dist-test-slave-qlq8"
server_key: "94e49e91f2d171f6d34bd29702dc31b5"
server_key_iv: "e562f8ec8a880df67737a754359db786"
server_key_version: "encryptionkey@0"
I20260926 07:38:07.452467 8942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "c106df996c6b46c58e77e91916611eca"
format_stamp: "Formatted at 2026-09-26 07:38:07 on dist-test-slave-qlq8"
server_key: "94e49e91f2d171f6d34bd29702dc31b5"
server_key_iv: "e562f8ec8a880df67737a754359db786"
server_key_version: "encryptionkey@0"
I20260926 07:38:07.457593 8942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:38:07.461313 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:07.462811 8942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:38:07.463006 8942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "c106df996c6b46c58e77e91916611eca"
format_stamp: "Formatted at 2026-09-26 07:38:07 on dist-test-slave-qlq8"
server_key: "94e49e91f2d171f6d34bd29702dc31b5"
server_key_iv: "e562f8ec8a880df67737a754359db786"
server_key_version: "encryptionkey@0"
I20260926 07:38:07.463271 8942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:07.492661 8942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:07.493464 8942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:07.493712 8942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:07.494719 8942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:07.496234 8942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:07.496320 8942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:07.496426 8942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:07.496517 8942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:07.529825 8942 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:37551
I20260926 07:38:07.529917 9068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:37551 every 8 connection(s)
I20260926 07:38:07.531150 8942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:38:07.537473 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 8942
I20260926 07:38:07.537664 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:38:07.538254 23413 external_mini_cluster.cc:1428] Setting key beceb4bbd8fb5bdcf961f8bd28f61b9f
I20260926 07:38:07.544587 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.533589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:38095 (local address 127.22.221.126:35945)
0926 07:38:07.533767 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.533771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.535502 (+ 1731us) server_negotiation.cc:407] Connection header received
0926 07:38:07.537806 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.537812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.537880 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.537974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:07.539373 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.540161 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.541693 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.542122 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.542256 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.543647 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:07.543652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:07.543655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:07.543673 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:07.543703 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:07.543717 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:07.543811 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.544165 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.544263 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:38:07.545540 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.534114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35945 (local address 127.22.221.67:38095)
0926 07:38:07.535339 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.535378 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.537461 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.538235 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.538249 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.538369 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:07.539132 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.539159 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.540318 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.540324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.541541 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.541553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.542357 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.542365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.542590 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.543338 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:07.543341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:07.543379 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:07.543452 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:07.543810 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:07.543822 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:07.543832 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.544145 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.544538 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":308,"threads_started":1}
I20260926 07:38:07.546829 9069 heartbeater.cc:347] Connected to a master server at 127.22.221.126:35945
I20260926 07:38:07.547092 9069 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:07.547793 9069 heartbeater.cc:511] Master 127.22.221.126:35945 requested a full tablet report, sending...
I20260926 07:38:07.549063 8622 ts_manager.cc:194] Registered new tserver with Master: c106df996c6b46c58e77e91916611eca (127.22.221.67:37551)
I20260926 07:38:07.549888 8622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.67:38095
I20260926 07:38:07.553298 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:38:07.563115 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.557513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.22.221.126:35945)
0926 07:38:07.557707 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.557711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.558211 (+ 500us) server_negotiation.cc:407] Connection header received
0926 07:38:07.558333 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.558336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.558420 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.558503 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:07.558886 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.559681 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.561110 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.561493 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.561592 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.562529 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:07.562534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:07.562537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:07.562553 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:07.562582 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:07.562594 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:07.562663 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.562790 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.562860 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:38:07.566658 8622 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:38:07.573937 8622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:38:07.611826 9081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.586071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:37551 (local address 127.0.0.1:41584)
0926 07:38:07.602032 (+ 15961us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.602086 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.602307 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.604760 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.604771 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.604817 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:07.605297 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.605309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.607213 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.607220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.608969 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.608988 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.610856 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.610862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.611056 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.611107 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.611285 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.611487 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15605,"thread_start_us":16471,"threads_started":1}
I20260926 07:38:07.611866 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.586421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.22.221.67:37551)
0926 07:38:07.603975 (+ 17554us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.603988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.604035 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:38:07.604165 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.604176 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.604503 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.604739 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:07.605514 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.607084 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.609202 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.610663 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.610813 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.611132 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.611316 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.611496 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17310,"thread_start_us":17225,"threads_started":1}
I20260926 07:38:07.613130 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.586164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43147 (local address 127.0.0.1:57102)
0926 07:38:07.605072 (+ 18908us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.605091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.605228 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.606137 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.606142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.606170 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:07.606623 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.606636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.608707 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.608714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.610517 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.610532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.612328 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.612334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.612509 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.612560 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.612641 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.612765 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18447,"mutex_wait_us":28,"spinlock_wait_cycles":6400,"thread_start_us":17119,"threads_started":1}
I20260926 07:38:07.613296 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.587224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.22.221.66:43147)
0926 07:38:07.603701 (+ 16477us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.603708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.605370 (+ 1662us) server_negotiation.cc:407] Connection header received
0926 07:38:07.605496 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.605504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.605739 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.605928 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:07.606798 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.608258 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.610708 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.612134 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.612275 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.612498 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.612783 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.612979 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16309,"thread_start_us":16201,"threads_started":1}
I20260926 07:38:07.614565 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.587916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35274)
0926 07:38:07.605095 (+ 17179us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.605116 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.605330 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.606039 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.606044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.606070 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:07.606441 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.606450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.608401 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.608406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.610663 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.610676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.613675 (+ 2999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.613683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.613966 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.613999 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.614101 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.614226 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16981,"mutex_wait_us":3,"thread_start_us":16832,"threads_started":1}
I20260926 07:38:07.614920 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.588538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.22.221.65:45407)
0926 07:38:07.595905 (+ 7367us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.595918 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.605222 (+ 9304us) server_negotiation.cc:407] Connection header received
0926 07:38:07.605526 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.605534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.605789 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.605964 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:07.606625 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.608258 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.610896 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.612360 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.612453 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.612664 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.614163 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.614329 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7144,"thread_start_us":7037,"threads_started":1}
I20260926 07:38:07.616923 9004 tablet_service.cc:1467] Processing CreateTablet for tablet 1caa2daa21934959a122868ddf770642 (DEFAULT_TABLE table=test-table [id=db5839515f324a82bccff9afa8ba20b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:07.616923 8874 tablet_service.cc:1467] Processing CreateTablet for tablet 1caa2daa21934959a122868ddf770642 (DEFAULT_TABLE table=test-table [id=db5839515f324a82bccff9afa8ba20b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:07.618284 8874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caa2daa21934959a122868ddf770642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:07.618351 9004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caa2daa21934959a122868ddf770642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:07.618191 8743 tablet_service.cc:1467] Processing CreateTablet for tablet 1caa2daa21934959a122868ddf770642 (DEFAULT_TABLE table=test-table [id=db5839515f324a82bccff9afa8ba20b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:07.619486 8743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caa2daa21934959a122868ddf770642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:07.631346 9087 tablet_bootstrap.cc:492] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Bootstrap starting.
I20260926 07:38:07.635560 9087 tablet_bootstrap.cc:654] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:07.636621 9087 log.cc:824] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:07.638442 9088 tablet_bootstrap.cc:492] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Bootstrap starting.
I20260926 07:38:07.639412 9087 tablet_bootstrap.cc:492] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: No bootstrap required, opened a new log
I20260926 07:38:07.639674 9087 ts_tablet_manager.cc:1397] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 07:38:07.641769 9088 tablet_bootstrap.cc:654] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:07.642469 8808 heartbeater.cc:503] Master 127.22.221.126:35945 was elected leader, sending a full tablet report...
I20260926 07:38:07.642913 9088 log.cc:824] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:07.643785 9089 tablet_bootstrap.cc:492] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: Bootstrap starting.
I20260926 07:38:07.647204 9089 tablet_bootstrap.cc:654] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:07.648013 9087 raft_consensus.cc:348] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.648248 9089 log.cc:824] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:07.648386 9087 raft_consensus.cc:374] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:07.648495 9087 raft_consensus.cc:729] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9172d8810cc648eda58e6d435b37882c, State: Initialized, Role: FOLLOWER
I20260926 07:38:07.649084 9087 consensus_queue.cc:260] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.650266 8939 heartbeater.cc:503] Master 127.22.221.126:35945 was elected leader, sending a full tablet report...
I20260926 07:38:07.650933 9087 ts_tablet_manager.cc:1428] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 07:38:07.651813 9088 tablet_bootstrap.cc:492] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: No bootstrap required, opened a new log
I20260926 07:38:07.652091 9088 ts_tablet_manager.cc:1397] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260926 07:38:07.654884 9089 tablet_bootstrap.cc:492] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: No bootstrap required, opened a new log
I20260926 07:38:07.655141 9089 ts_tablet_manager.cc:1397] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260926 07:38:07.659750 9088 raft_consensus.cc:348] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.660122 9088 raft_consensus.cc:374] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:07.660226 9088 raft_consensus.cc:729] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106df996c6b46c58e77e91916611eca, State: Initialized, Role: FOLLOWER
I20260926 07:38:07.660787 9088 consensus_queue.cc:260] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.662340 9089 raft_consensus.cc:348] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.662693 9089 raft_consensus.cc:374] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:07.662786 9089 raft_consensus.cc:729] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeef545cc684d98a8bde4d9e7f63d6b, State: Initialized, Role: FOLLOWER
I20260926 07:38:07.663414 9089 consensus_queue.cc:260] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.665164 9089 ts_tablet_manager.cc:1428] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 07:38:07.666033 9069 heartbeater.cc:503] Master 127.22.221.126:35945 was elected leader, sending a full tablet report...
I20260926 07:38:07.666711 9088 ts_tablet_manager.cc:1428] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260926 07:38:07.708971 9094 raft_consensus.cc:482] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:07.709293 9094 raft_consensus.cc:504] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.711427 9094 leader_election.cc:288] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9172d8810cc648eda58e6d435b37882c (127.22.221.66:43147), 0eeef545cc684d98a8bde4d9e7f63d6b (127.22.221.65:45407)
I20260926 07:38:07.716944 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.711700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:39285 (local address 127.22.221.66:43147)
0926 07:38:07.711909 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.711915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.712023 (+ 108us) server_negotiation.cc:407] Connection header received
0926 07:38:07.712243 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.712251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.712356 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.712466 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:07.712968 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.713683 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.715339 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.716078 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.716239 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.716308 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.716568 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.716692 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:38:07.716967 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.711607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43147 (local address 127.22.221.67:39285)
0926 07:38:07.711910 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.711926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.712090 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.712492 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.712497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.712533 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:07.712809 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.712819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.713810 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.713818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.715158 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.715170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.716229 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.716238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.716403 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.716434 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.716541 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.716681 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 07:38:07.717685 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.712320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:42665 (local address 127.22.221.65:45407)
0926 07:38:07.712491 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.712496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.712778 (+ 282us) server_negotiation.cc:407] Connection header received
0926 07:38:07.712965 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.712969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.713038 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.713131 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:07.713672 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.714323 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.716042 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.716773 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.716942 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.717007 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.717317 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.717448 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:38:07.717744 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.711901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.22.221.67:42665)
0926 07:38:07.712674 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.712688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:07.712823 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:07.713157 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:07.713161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:07.713184 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:07.713512 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.713520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.714469 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.714474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.715876 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:07.715887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.716946 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.716952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:07.717151 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.717188 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:38:07.717292 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:38:07.717406 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":1,"thread_start_us":595,"threads_started":1}
I20260926 07:38:07.717811 8894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caa2daa21934959a122868ddf770642" candidate_uuid: "c106df996c6b46c58e77e91916611eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9172d8810cc648eda58e6d435b37882c" is_pre_election: true
I20260926 07:38:07.718369 8894 raft_consensus.cc:2454] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c106df996c6b46c58e77e91916611eca in term 0.
I20260926 07:38:07.718918 8763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caa2daa21934959a122868ddf770642" candidate_uuid: "c106df996c6b46c58e77e91916611eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" is_pre_election: true
I20260926 07:38:07.719070 8957 leader_election.cc:302] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9172d8810cc648eda58e6d435b37882c, c106df996c6b46c58e77e91916611eca; no voters:
I20260926 07:38:07.719449 9094 raft_consensus.cc:2781] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:38:07.719475 8763 raft_consensus.cc:2454] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c106df996c6b46c58e77e91916611eca in term 0.
I20260926 07:38:07.719619 9094 raft_consensus.cc:482] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:38:07.719728 9094 raft_consensus.cc:3037] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:07.722035 9094 raft_consensus.cc:504] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.722656 9094 leader_election.cc:288] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [CANDIDATE]: Term 1 election: Requested vote from peers 9172d8810cc648eda58e6d435b37882c (127.22.221.66:43147), 0eeef545cc684d98a8bde4d9e7f63d6b (127.22.221.65:45407)
I20260926 07:38:07.723146 8894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caa2daa21934959a122868ddf770642" candidate_uuid: "c106df996c6b46c58e77e91916611eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9172d8810cc648eda58e6d435b37882c"
I20260926 07:38:07.723147 8763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caa2daa21934959a122868ddf770642" candidate_uuid: "c106df996c6b46c58e77e91916611eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b"
I20260926 07:38:07.723379 8763 raft_consensus.cc:3037] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:07.723373 8894 raft_consensus.cc:3037] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:07.725550 8894 raft_consensus.cc:2454] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c106df996c6b46c58e77e91916611eca in term 1.
I20260926 07:38:07.725818 8763 raft_consensus.cc:2454] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c106df996c6b46c58e77e91916611eca in term 1.
I20260926 07:38:07.725914 8957 leader_election.cc:302] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9172d8810cc648eda58e6d435b37882c, c106df996c6b46c58e77e91916611eca; no voters:
I20260926 07:38:07.726214 9094 raft_consensus.cc:2781] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:07.726650 9094 raft_consensus.cc:686] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 1 LEADER]: Becoming Leader. State: Replica: c106df996c6b46c58e77e91916611eca, State: Running, Role: LEADER
I20260926 07:38:07.727159 9094 consensus_queue.cc:237] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } }
I20260926 07:38:07.732095 8621 catalog_manager.cc:5495] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca reported cstate change: term changed from 0 to 1, leader changed from <none> to c106df996c6b46c58e77e91916611eca (127.22.221.67). New cstate: current_term: 1 leader_uuid: "c106df996c6b46c58e77e91916611eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106df996c6b46c58e77e91916611eca" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 37551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 } health_report { overall_health: UNKNOWN } } }
I20260926 07:38:07.777441 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.772056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.22.221.67:37551)
0926 07:38:07.772283 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.772290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.772307 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:38:07.772399 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.772403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.772475 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.772653 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:38:07.773036 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.773747 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.775206 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.775693 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.775826 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.776333 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:38:07.776853 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:38:07.777044 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.777101 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.777181 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 07:38:07.782759 9070 tablet.cc:2369] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:07.786276 8894 raft_consensus.cc:1261] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 1 FOLLOWER]: Refusing update from remote peer c106df996c6b46c58e77e91916611eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:07.786276 8763 raft_consensus.cc:1261] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 1 FOLLOWER]: Refusing update from remote peer c106df996c6b46c58e77e91916611eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:07.787586 9094 consensus_queue.cc:1035] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9172d8810cc648eda58e6d435b37882c" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:07.787979 9097 consensus_queue.cc:1035] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eeef545cc684d98a8bde4d9e7f63d6b" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:07.798058 9100 mvcc.cc:205] Tried to move back new op lower bound from 7333512346761433088 to 7333512346534981632. Current Snapshot: MvccSnapshot[applied={T|T < 7333512346761433088}]
I20260926 07:38:07.816609 8621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:07.816906 8621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:07.820575 8621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e61bb1685c4e42d1ab608b5bbbd45ab5: Sending DeleteTablet for 3 replicas of tablet 1caa2daa21934959a122868ddf770642
I20260926 07:38:07.821867 8874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caa2daa21934959a122868ddf770642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:07 UTC) from {username='slave'} at 127.0.0.1:57102
I20260926 07:38:07.822003 9004 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caa2daa21934959a122868ddf770642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:07 UTC) from {username='slave'} at 127.0.0.1:41584
I20260926 07:38:07.822053 8743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caa2daa21934959a122868ddf770642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:07 UTC) from {username='slave'} at 127.0.0.1:35274
I20260926 07:38:07.824146 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8681
I20260926 07:38:07.832929 9111 tablet_replica.cc:330] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: stopping tablet replica
I20260926 07:38:07.833463 9111 raft_consensus.cc:2229] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:38:07.834254 9111 raft_consensus.cc:2258] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:07.839147 9111 ts_tablet_manager.cc:1905] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:07.845554 9112 tablet_replica.cc:330] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b: stopping tablet replica
I20260926 07:38:07.847028 9111 ts_tablet_manager.cc:1918] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:07.847198 9111 log.cc:1196] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/1caa2daa21934959a122868ddf770642
I20260926 07:38:07.847831 9111 ts_tablet_manager.cc:1939] T 1caa2daa21934959a122868ddf770642 P c106df996c6b46c58e77e91916611eca: Deleting consensus metadata
I20260926 07:38:07.848237 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.825981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.22.221.65:45407)
0926 07:38:07.826219 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.826229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.838736 (+ 12507us) server_negotiation.cc:407] Connection header received
0926 07:38:07.839213 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.839219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.839359 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.839436 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:07.840193 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.841427 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.843429 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.844134 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.844242 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.846137 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:07.846148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:07.846165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:07.847083 (+ 918us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:07.847144 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:07.847169 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:07.847417 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.847694 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.847821 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:38:07.849581 8803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:07.851168 8609 catalog_manager.cc:4841] TS c106df996c6b46c58e77e91916611eca (127.22.221.67:37551): tablet 1caa2daa21934959a122868ddf770642 (table test-table [id=db5839515f324a82bccff9afa8ba20b1]) successfully deleted
I20260926 07:38:07.851294 9110 tablet_replica.cc:330] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: stopping tablet replica
I20260926 07:38:07.851984 9110 raft_consensus.cc:2229] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:07.852692 9110 raft_consensus.cc:2258] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:07.855122 9110 ts_tablet_manager.cc:1905] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:07.859903 9110 ts_tablet_manager.cc:1918] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:07.860075 9110 log.cc:1196] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/1caa2daa21934959a122868ddf770642
I20260926 07:38:07.860579 9110 ts_tablet_manager.cc:1939] T 1caa2daa21934959a122868ddf770642 P 9172d8810cc648eda58e6d435b37882c: Deleting consensus metadata
I20260926 07:38:07.862140 8606 catalog_manager.cc:4841] TS 9172d8810cc648eda58e6d435b37882c (127.22.221.66:43147): tablet 1caa2daa21934959a122868ddf770642 (table test-table [id=db5839515f324a82bccff9afa8ba20b1]) successfully deleted
I20260926 07:38:07.849972 9112 raft_consensus.cc:2229] T 1caa2daa21934959a122868ddf770642 P 0eeef545cc684d98a8bde4d9e7f63d6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:07.965478 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8681
W20260926 07:38:07.983155 8607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.221.65:45407 (error 108)
I20260926 07:38:07.983933 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8812
I20260926 07:38:07.985471 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.984916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35288)
0926 07:38:07.985166 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:07.985289 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260926 07:38:07.985731 8607 catalog_manager.cc:4584] TS 0eeef545cc684d98a8bde4d9e7f63d6b (127.22.221.65:45407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1caa2daa21934959a122868ddf770642: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
I20260926 07:38:07.991391 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:07.984825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.22.221.66:43147)
0926 07:38:07.985006 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:07.985013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:07.985039 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:38:07.985262 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:07.985266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:07.985348 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:07.985461 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:07.985909 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.986650 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.988054 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:07.988459 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:07.988591 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:07.989848 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:07.989858 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:07.989871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:07.990558 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:07.990606 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:07.990626 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:07.990763 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:07.990898 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:38:07.991006 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:38:07.992046 8934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:08.039757 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.039089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35292)
0926 07:38:08.039431 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.039552 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 07:38:08.041163 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.040700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35296)
0926 07:38:08.040906 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.040994 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 07:38:08.099794 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8812
I20260926 07:38:08.102617 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.102032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35304)
0926 07:38:08.102290 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.102407 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:38:08.104008 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.103547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35306)
0926 07:38:08.103741 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.103838 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:38:08.118127 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8942
I20260926 07:38:08.126247 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.118960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.22.221.67:37551)
0926 07:38:08.119134 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:08.119138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:08.119196 (+ 58us) server_negotiation.cc:407] Connection header received
0926 07:38:08.119436 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:08.119440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:08.119559 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:08.119647 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:08.120163 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:08.120981 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:08.122947 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:08.123460 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:08.123579 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:08.124802 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:08.124810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:08.124825 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:08.125454 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:08.125505 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:08.125524 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:08.125602 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:08.125787 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 07:38:08.125868 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:38:08.126884 9064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:08.184592 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.184016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35322)
0926 07:38:08.184267 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.184398 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 07:38:08.185743 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.185293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35338)
0926 07:38:08.185518 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.185607 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 07:38:08.249611 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8942
I20260926 07:38:08.268997 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 8591
I20260926 07:38:08.270244 8652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:08.400928 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:08.398564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45407 (local address 127.0.0.1:35354)
0926 07:38:08.399184 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:08.399339 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.65:45407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260926 07:38:08.401698 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 8591
2026-09-26T07:38:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2878 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:08 dist-test-slave-qlq8 krb5kdc[9121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:08 dist-test-slave-qlq8 krb5kdc[9121](info): set up 2 sockets
Sep 26 07:38:08 dist-test-slave-qlq8 krb5kdc[9121](info): commencing operation
krb5kdc: starting...
W20260926 07:38:09.527654 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:09 dist-test-slave-qlq8 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:09.786945 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:44917
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:10.115015 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:10.115397 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:10.115566 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:10.122902 9134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:10.123059 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:10.123184 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:10.123315 9134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:10.123407 9134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:10.133353 9134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:10.134750 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:10.136122 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:10.147985 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:10.147933 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:10.148022 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:10.149127 9134 server_base.cc:1034] running on GCE node
I20260926 07:38:10.149981 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:10.151564 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:10.152807 9134 hybrid_clock.cc:648] HybridClock initialized: now 1790408290152764 us; error 49 us; skew 500 ppm
I20260926 07:38:10.153237 9134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:10 dist-test-slave-qlq8 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408290, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:10.167346 9134 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:10.169116 9134 webserver.cc:460] Webserver started at http://127.22.221.126:41897/ using document root <none> and password file <none>
I20260926 07:38:10.169909 9134 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:10.170071 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:10.170403 9134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:10.173704 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "d2ac6e16cba248338306556449ebf125"
format_stamp: "Formatted at 2026-09-26 07:38:10 on dist-test-slave-qlq8"
server_key: "8216814082c5a30d50e512ceb29fc553"
server_key_iv: "5bedcfe569324c61b563057bbc44e6a0"
server_key_version: "encryptionkey@0"
I20260926 07:38:10.174589 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "d2ac6e16cba248338306556449ebf125"
format_stamp: "Formatted at 2026-09-26 07:38:10 on dist-test-slave-qlq8"
server_key: "8216814082c5a30d50e512ceb29fc553"
server_key_iv: "5bedcfe569324c61b563057bbc44e6a0"
server_key_version: "encryptionkey@0"
I20260926 07:38:10.180209 9134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:38:10.183837 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:10.185266 9134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:38:10.185474 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "d2ac6e16cba248338306556449ebf125"
format_stamp: "Formatted at 2026-09-26 07:38:10 on dist-test-slave-qlq8"
server_key: "8216814082c5a30d50e512ceb29fc553"
server_key_iv: "5bedcfe569324c61b563057bbc44e6a0"
server_key_version: "encryptionkey@0"
I20260926 07:38:10.185679 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:10.207461 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:10.211699 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:10.212007 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:10.230386 9134 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:44917
I20260926 07:38:10.230408 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:44917 every 8 connection(s)
I20260926 07:38:10.231788 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:10.235462 9201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:10.242179 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9134
I20260926 07:38:10.242379 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:10.242949 23413 external_mini_cluster.cc:1428] Setting key a83cab6aa8ef89277acf38e498b5ef79
I20260926 07:38:10.246402 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125: Bootstrap starting.
I20260926 07:38:10.250159 9201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:10.251406 9201 log.cc:824] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:10.254508 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125: No bootstrap required, opened a new log
Sep 26 07:38:10 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:10.263440 9201 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac6e16cba248338306556449ebf125" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 44917 } }
I20260926 07:38:10.263854 9201 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:10.263978 9201 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ac6e16cba248338306556449ebf125, State: Initialized, Role: FOLLOWER
I20260926 07:38:10.264708 9201 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac6e16cba248338306556449ebf125" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 44917 } }
I20260926 07:38:10.265022 9201 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:10.265196 9201 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:10.265380 9201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:10.267797 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:10.244854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.22.221.126:44917)
0926 07:38:10.245653 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:10.245664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:10.245705 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:38:10.247523 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:10.247547 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:10.247941 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:10.248289 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:10.250293 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:10.251785 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:10.253661 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:10.254305 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:10.254441 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:10.259412 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:10.259426 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:10.259441 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:10.259499 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:10.262719 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:10.263552 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:10.263560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:10.263567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:10.263734 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:10.264187 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:10.264192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:10.264195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:10.264937 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:10.265253 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:10.266351 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0926 07:38:10.266695 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":382,"threads_started":1}
I20260926 07:38:10.267820 9201 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac6e16cba248338306556449ebf125" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 44917 } }
I20260926 07:38:10.268371 9201 leader_election.cc:302] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ac6e16cba248338306556449ebf125; no voters:
I20260926 07:38:10.268998 9201 leader_election.cc:288] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:10.269151 9206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:10.270416 9206 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [term 1 LEADER]: Becoming Leader. State: Replica: d2ac6e16cba248338306556449ebf125, State: Running, Role: LEADER
I20260926 07:38:10.271031 9206 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac6e16cba248338306556449ebf125" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 44917 } }
I20260926 07:38:10.271277 9201 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:10.274354 9207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ac6e16cba248338306556449ebf125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac6e16cba248338306556449ebf125" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 44917 } } }
I20260926 07:38:10.274329 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ac6e16cba248338306556449ebf125. Latest consensus state: current_term: 1 leader_uuid: "d2ac6e16cba248338306556449ebf125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac6e16cba248338306556449ebf125" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 44917 } } }
I20260926 07:38:10.275408 9207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:10.275416 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:10.277129 9213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:10.282961 9213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:10.292951 9213 catalog_manager.cc:1347] Generated new cluster ID: e68cca8dd0264531a7cc2935aa2e1462
I20260926 07:38:10.293118 9213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:10.305764 9213 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:10.307189 9213 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:10.318095 9213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125: Generated new TSK 0
I20260926 07:38:10.319213 9213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:38:10.386494 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:44917
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:10.723166 9227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:10.723578 9227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:10.723759 9227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:10.731040 9227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:10.731210 9227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:10.731389 9227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:38:10.741469 9227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:44917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:10.742854 9227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:10.744318 9227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:10.758098 9232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:10.758049 9235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:10.758117 9233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:10.764365 9227 server_base.cc:1034] running on GCE node
I20260926 07:38:10.765125 9227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:10.766144 9227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:10.767336 9227 hybrid_clock.cc:648] HybridClock initialized: now 1790408290767263 us; error 75 us; skew 500 ppm
I20260926 07:38:10.767666 9227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:10 dist-test-slave-qlq8 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408290, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:10.770753 9227 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:38:10.772454 9227 webserver.cc:460] Webserver started at http://127.22.221.65:33589/ using document root <none> and password file <none>
I20260926 07:38:10.773178 9227 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:10.773321 9227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:10.773635 9227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:10.776623 9227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "e12ab23573ed437a9954d39f07da75b4"
format_stamp: "Formatted at 2026-09-26 07:38:10 on dist-test-slave-qlq8"
server_key: "7453221d1784079679e17302aa63a3b8"
server_key_iv: "f1b43e81d59ddcf85e712572d0ceef80"
server_key_version: "encryptionkey@0"
I20260926 07:38:10.777513 9227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "e12ab23573ed437a9954d39f07da75b4"
format_stamp: "Formatted at 2026-09-26 07:38:10 on dist-test-slave-qlq8"
server_key: "7453221d1784079679e17302aa63a3b8"
server_key_iv: "f1b43e81d59ddcf85e712572d0ceef80"
server_key_version: "encryptionkey@0"
I20260926 07:38:10.782846 9227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:10.786440 9242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:10.787842 9227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:38:10.788018 9227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "e12ab23573ed437a9954d39f07da75b4"
format_stamp: "Formatted at 2026-09-26 07:38:10 on dist-test-slave-qlq8"
server_key: "7453221d1784079679e17302aa63a3b8"
server_key_iv: "f1b43e81d59ddcf85e712572d0ceef80"
server_key_version: "encryptionkey@0"
I20260926 07:38:10.788226 9227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:10.800547 9227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:10.804419 9227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:10.804708 9227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:10.805608 9227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:10.806885 9227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:10.806991 9227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:10.807147 9227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:10.807220 9227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:10.840975 9227 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:45729
I20260926 07:38:10.841037 9354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:45729 every 8 connection(s)
I20260926 07:38:10.842327 9227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:38:10.844985 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9227
I20260926 07:38:10.845216 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:38:10.845765 23413 external_mini_cluster.cc:1428] Setting key 5e7908373dae2dbc53cb592880498992
Sep 26 07:38:10 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408290, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:10.865609 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:10.845008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:54025 (local address 127.22.221.126:44917)
0926 07:38:10.845542 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:10.845557 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:10.846585 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 07:38:10.849086 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:10.849091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:10.849162 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:10.849271 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:10.851568 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:10.852354 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:10.855075 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:10.855567 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:10.855677 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:10.859728 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:10.859732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:10.859736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:10.859771 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:10.862453 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:10.863193 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:10.863198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:10.863201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:10.863374 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:10.863819 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:10.863824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:10.863827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:10.864111 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:10.864269 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:10.864994 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 07:38:10.865234 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260926 07:38:10.866580 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:10.845556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:44917 (local address 127.22.221.65:54025)
0926 07:38:10.846384 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:10.846448 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:10.848744 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:10.849596 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:10.849610 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:10.850442 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:10.851367 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:10.851397 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:10.852551 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:10.852560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:10.854887 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:10.854914 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:10.855819 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:10.855828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:10.856178 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:10.856969 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:10.857004 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:10.859473 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:10.862603 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:10.862612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:10.862630 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:10.863053 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:10.863490 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:10.863494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:10.863498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:10.863693 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:10.864243 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:10.864251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:10.864602 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 07:38:10.864871 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 07:38:10.865412 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":330,"threads_started":1}
I20260926 07:38:10.868491 9355 heartbeater.cc:347] Connected to a master server at 127.22.221.126:44917
I20260926 07:38:10.868809 9355 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:10.869511 9355 heartbeater.cc:511] Master 127.22.221.126:44917 requested a full tablet report, sending...
I20260926 07:38:10.872141 9166 ts_manager.cc:194] Registered new tserver with Master: e12ab23573ed437a9954d39f07da75b4 (127.22.221.65:45729)
I20260926 07:38:10.874634 9166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:54025
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:38:10.906797 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:44917
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:11.234272 9361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:11.234525 9361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:11.234679 9361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:11.241979 9361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:11.242076 9361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:11.242235 9361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:38:11.251856 9361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:44917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:11.253079 9361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:11.254333 9361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:11.267294 9366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:11.267252 9369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:11.267295 9367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:11.268816 9361 server_base.cc:1034] running on GCE node
I20260926 07:38:11.269339 9361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:11.270260 9361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:11.271448 9361 hybrid_clock.cc:648] HybridClock initialized: now 1790408291271393 us; error 40 us; skew 500 ppm
I20260926 07:38:11.271765 9361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:11 dist-test-slave-qlq8 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408291, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:11.274902 9361 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:38:11.276392 9361 webserver.cc:460] Webserver started at http://127.22.221.66:33089/ using document root <none> and password file <none>
I20260926 07:38:11.277166 9361 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:11.277288 9361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:11.277599 9361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:11.280674 9361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "182ccf9aff2f4f1990423e4c8c328bf3"
format_stamp: "Formatted at 2026-09-26 07:38:11 on dist-test-slave-qlq8"
server_key: "192bd7814dd356c02fb6060e6e643d02"
server_key_iv: "aff36fea861c76fc1da6b45ec24476e6"
server_key_version: "encryptionkey@0"
I20260926 07:38:11.281456 9361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "182ccf9aff2f4f1990423e4c8c328bf3"
format_stamp: "Formatted at 2026-09-26 07:38:11 on dist-test-slave-qlq8"
server_key: "192bd7814dd356c02fb6060e6e643d02"
server_key_iv: "aff36fea861c76fc1da6b45ec24476e6"
server_key_version: "encryptionkey@0"
I20260926 07:38:11.287318 9361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:38:11.291213 9376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:11.292826 9361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:38:11.293057 9361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "182ccf9aff2f4f1990423e4c8c328bf3"
format_stamp: "Formatted at 2026-09-26 07:38:11 on dist-test-slave-qlq8"
server_key: "192bd7814dd356c02fb6060e6e643d02"
server_key_iv: "aff36fea861c76fc1da6b45ec24476e6"
server_key_version: "encryptionkey@0"
I20260926 07:38:11.293279 9361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:11.311313 9361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:11.315029 9361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:11.315342 9361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:11.316305 9361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:11.317626 9361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:11.317700 9361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:11.317842 9361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:11.317901 9361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:11.352345 9361 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:43429
I20260926 07:38:11.352469 9488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:43429 every 8 connection(s)
I20260926 07:38:11.353662 9361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:38:11.361294 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9361
I20260926 07:38:11.361485 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:38:11.361987 23413 external_mini_cluster.cc:1428] Setting key 3301fdab67f97cea059c2c24444e1728
Sep 26 07:38:11 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408291, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:11.374842 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.356249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:38721 (local address 127.22.221.126:44917)
0926 07:38:11.356426 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:11.356430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:11.357481 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 07:38:11.359296 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:11.359303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:11.359401 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:11.359515 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:11.360990 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.361804 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.363591 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.364126 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.364255 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.367831 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:11.367839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:11.367847 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:11.367889 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:11.371562 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:11.372269 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:11.372277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:11.372283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:11.372482 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:11.372968 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:11.372975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:11.372981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:11.373359 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:11.373522 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:11.374417 (+ 895us) server_negotiation.cc:299] Negotiation successful
0926 07:38:11.374597 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:38:11.376368 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.356627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:44917 (local address 127.22.221.66:38721)
0926 07:38:11.357312 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:11.357352 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:11.359006 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:11.359746 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:11.359754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:11.360235 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:11.360777 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.360794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.361965 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.361969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.363442 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.363454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.364341 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.364347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.364547 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.365239 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:11.365260 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:11.367599 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:11.371687 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:11.371693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:11.371704 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:11.372148 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:11.372588 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:11.372592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:11.372594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:11.372864 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:11.373498 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:11.373508 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:11.373961 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 07:38:11.374336 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 07:38:11.374921 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":271,"threads_started":1}
I20260926 07:38:11.377820 9489 heartbeater.cc:347] Connected to a master server at 127.22.221.126:44917
I20260926 07:38:11.378206 9489 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:11.378991 9489 heartbeater.cc:511] Master 127.22.221.126:44917 requested a full tablet report, sending...
I20260926 07:38:11.380679 9166 ts_manager.cc:194] Registered new tserver with Master: 182ccf9aff2f4f1990423e4c8c328bf3 (127.22.221.66:43429)
I20260926 07:38:11.381517 9166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:38721
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:38:11.420482 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:44917
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:11.749159 9494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:11.749440 9494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:11.749600 9494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:11.757558 9494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:11.757673 9494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:11.757858 9494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:38:11.767956 9494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:44917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:11.769253 9494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:11.770574 9494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:11.784482 9499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:11.784508 9500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:11.784548 9502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:11.785658 9494 server_base.cc:1034] running on GCE node
I20260926 07:38:11.786382 9494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:11.787436 9494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:11.788708 9494 hybrid_clock.cc:648] HybridClock initialized: now 1790408291788651 us; error 60 us; skew 500 ppm
I20260926 07:38:11.789189 9494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:11 dist-test-slave-qlq8 krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408291, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:11.793047 9494 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:38:11.794648 9494 webserver.cc:460] Webserver started at http://127.22.221.67:41629/ using document root <none> and password file <none>
I20260926 07:38:11.795431 9494 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:11.795600 9494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:11.795928 9494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:11.798957 9494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "7792756660304b58951426926d5e5879"
format_stamp: "Formatted at 2026-09-26 07:38:11 on dist-test-slave-qlq8"
server_key: "8a4d194dc1c7b23d0adf3b1f2c5faa57"
server_key_iv: "d43f079a80cab9d056325cd1635b5e99"
server_key_version: "encryptionkey@0"
I20260926 07:38:11.799782 9494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "7792756660304b58951426926d5e5879"
format_stamp: "Formatted at 2026-09-26 07:38:11 on dist-test-slave-qlq8"
server_key: "8a4d194dc1c7b23d0adf3b1f2c5faa57"
server_key_iv: "d43f079a80cab9d056325cd1635b5e99"
server_key_version: "encryptionkey@0"
I20260926 07:38:11.805342 9494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:38:11.808848 9509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:11.810143 9494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:38:11.810319 9494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "7792756660304b58951426926d5e5879"
format_stamp: "Formatted at 2026-09-26 07:38:11 on dist-test-slave-qlq8"
server_key: "8a4d194dc1c7b23d0adf3b1f2c5faa57"
server_key_iv: "d43f079a80cab9d056325cd1635b5e99"
server_key_version: "encryptionkey@0"
I20260926 07:38:11.810518 9494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:11.839177 9494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:11.843952 9494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:11.844357 9494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:11.845403 9494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:11.846756 9494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:11.846829 9494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:11.846987 9494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:11.847056 9494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:11.878325 9355 heartbeater.cc:503] Master 127.22.221.126:44917 was elected leader, sending a full tablet report...
I20260926 07:38:11.880801 9494 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:33549
I20260926 07:38:11.880905 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:33549 every 8 connection(s)
I20260926 07:38:11.882104 9494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:38:11.885468 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9494
I20260926 07:38:11.885671 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:38:11.886253 23413 external_mini_cluster.cc:1428] Setting key a0673367ebed981720f511350675807d
Sep 26 07:38:11 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408291, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:11.902423 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.884586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:46825 (local address 127.22.221.126:44917)
0926 07:38:11.885021 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:11.885025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:11.886035 (+ 1010us) server_negotiation.cc:407] Connection header received
0926 07:38:11.888254 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:11.888259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:11.888356 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:11.888470 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:11.890195 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.891052 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.892733 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.893143 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.893274 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.896634 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:11.896638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:11.896641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:11.896693 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:11.899121 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:11.899975 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:11.899980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:11.899982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:11.900095 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:11.900685 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:11.900690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:11.900693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:11.901052 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:11.901236 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:11.901953 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 07:38:11.902159 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
I20260926 07:38:11.903396 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.885059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:44917 (local address 127.22.221.67:46825)
0926 07:38:11.885891 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:11.885928 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:11.887937 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:11.888709 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:11.888717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:11.889391 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:11.890025 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.890050 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.891214 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.891218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.892586 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.892597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.893379 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.893386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.893617 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.894240 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:11.894261 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:11.896413 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:11.899277 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:11.899286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:11.899302 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:11.899845 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:11.900315 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:11.900318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:11.900321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:11.900534 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:11.901238 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:11.901246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:11.901613 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 07:38:11.901837 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:38:11.902245 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":320,"threads_started":1}
I20260926 07:38:11.904680 9622 heartbeater.cc:347] Connected to a master server at 127.22.221.126:44917
I20260926 07:38:11.904985 9622 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:11.905699 9622 heartbeater.cc:511] Master 127.22.221.126:44917 requested a full tablet report, sending...
I20260926 07:38:11.907161 9166 ts_manager.cc:194] Registered new tserver with Master: 7792756660304b58951426926d5e5879 (127.22.221.67:33549)
I20260926 07:38:11.908006 9166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:46825
I20260926 07:38:11.913640 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:38:11.931460 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.918098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.22.221.126:44917)
0926 07:38:11.918259 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:11.918270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:11.918775 (+ 505us) server_negotiation.cc:407] Connection header received
0926 07:38:11.918909 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:11.918913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:11.918992 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:11.919074 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:11.920592 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.921304 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.922683 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.923081 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.923171 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.925968 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:11.925973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:11.925976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:11.926007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:11.928562 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:11.929424 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:11.929428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:11.929431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:11.929559 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:11.930002 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:11.930006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:11.930008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:11.930364 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:11.930585 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:11.930954 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 07:38:11.931129 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:38:11.935920 9166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:38:11.944087 9166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:38:11.970945 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.959267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43429 (local address 127.0.0.1:57938)
0926 07:38:11.960664 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:11.960717 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:11.960905 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:11.961951 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:11.961962 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:11.962005 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:11.962568 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.962583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.964768 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.964773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.967567 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.967583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.969884 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.969890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.970120 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.970160 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:38:11.970310 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 07:38:11.970513 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":975,"threads_started":1}
I20260926 07:38:11.970901 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.959494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.22.221.66:43429)
0926 07:38:11.960147 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:11.960157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:11.961116 (+ 959us) server_negotiation.cc:407] Connection header received
0926 07:38:11.961240 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:11.961249 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:11.961537 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:11.961748 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:11.962785 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.964558 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.967766 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.969436 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.969608 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.969842 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:11.970374 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 07:38:11.970550 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":341,"threads_started":1}
I20260926 07:38:11.973023 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.963395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.22.221.67:33549)
0926 07:38:11.964367 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:11.964375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:11.964408 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:38:11.964507 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:11.964514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:11.964754 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:11.965006 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:11.965689 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.967141 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.970100 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.971711 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.971866 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.972133 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:11.972467 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 07:38:11.972638 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":636,"threads_started":1}
I20260926 07:38:11.973050 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.962713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:33549 (local address 127.0.0.1:58654)
0926 07:38:11.963580 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:11.963599 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:11.963731 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:11.965010 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:11.965021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:11.965049 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:11.965497 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.965508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.967349 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.967355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.969641 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.969656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.971946 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.971954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.972176 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.972281 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:38:11.972405 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:38:11.972559 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":35,"thread_start_us":985,"threads_started":1}
I20260926 07:38:11.973140 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.959475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:45729 (local address 127.0.0.1:45718)
0926 07:38:11.962472 (+ 2997us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:11.962497 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:11.962626 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:11.963904 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:11.963910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:11.963940 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:11.964311 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.964321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.966679 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.966685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.969491 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:11.969509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.972079 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.972089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:11.972351 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.972411 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:38:11.972488 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:38:11.972632 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2807,"mutex_wait_us":71,"thread_start_us":2639,"threads_started":1}
I20260926 07:38:11.974660 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:11.960110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.22.221.65:45729)
0926 07:38:11.963279 (+ 3169us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:11.963288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:11.963318 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:38:11.963439 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:11.963447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:11.963676 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:11.964536 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:11.964768 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.966523 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.969701 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:11.971812 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:11.973757 (+ 1945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:11.974059 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:11.974146 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 07:38:11.974330 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2958,"thread_start_us":2783,"threads_started":1}
I20260926 07:38:11.975729 9424 tablet_service.cc:1467] Processing CreateTablet for tablet d93d7e5b181249b7897ce72612000a5e (DEFAULT_TABLE table=test-table [id=1251078cbac6482aae008d5faba470bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:11.977324 9424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93d7e5b181249b7897ce72612000a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:11.977521 9557 tablet_service.cc:1467] Processing CreateTablet for tablet d93d7e5b181249b7897ce72612000a5e (DEFAULT_TABLE table=test-table [id=1251078cbac6482aae008d5faba470bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:11.977805 9290 tablet_service.cc:1467] Processing CreateTablet for tablet d93d7e5b181249b7897ce72612000a5e (DEFAULT_TABLE table=test-table [id=1251078cbac6482aae008d5faba470bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:11.979074 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93d7e5b181249b7897ce72612000a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:11.979337 9290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93d7e5b181249b7897ce72612000a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:11.991413 9641 tablet_bootstrap.cc:492] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Bootstrap starting.
I20260926 07:38:11.993530 9642 tablet_bootstrap.cc:492] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Bootstrap starting.
I20260926 07:38:11.995800 9643 tablet_bootstrap.cc:492] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Bootstrap starting.
I20260926 07:38:11.997769 9642 tablet_bootstrap.cc:654] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:11.997769 9641 tablet_bootstrap.cc:654] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:11.999141 9643 tablet_bootstrap.cc:654] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:11.999329 9641 log.cc:824] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:11.999341 9642 log.cc:824] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:12.000547 9643 log.cc:824] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:12.003006 9642 tablet_bootstrap.cc:492] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: No bootstrap required, opened a new log
I20260926 07:38:12.003006 9641 tablet_bootstrap.cc:492] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: No bootstrap required, opened a new log
I20260926 07:38:12.003413 9641 ts_tablet_manager.cc:1397] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260926 07:38:12.003892 9643 tablet_bootstrap.cc:492] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: No bootstrap required, opened a new log
I20260926 07:38:12.003885 9642 ts_tablet_manager.cc:1397] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260926 07:38:12.004179 9643 ts_tablet_manager.cc:1397] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 07:38:12.011827 9643 raft_consensus.cc:348] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.012252 9643 raft_consensus.cc:374] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:12.012346 9643 raft_consensus.cc:729] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792756660304b58951426926d5e5879, State: Initialized, Role: FOLLOWER
I20260926 07:38:12.013012 9643 consensus_queue.cc:260] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.015790 9622 heartbeater.cc:503] Master 127.22.221.126:44917 was elected leader, sending a full tablet report...
I20260926 07:38:12.015604 9641 raft_consensus.cc:348] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.016068 9641 raft_consensus.cc:374] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:12.016171 9641 raft_consensus.cc:729] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12ab23573ed437a9954d39f07da75b4, State: Initialized, Role: FOLLOWER
I20260926 07:38:12.016644 9642 raft_consensus.cc:348] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.016947 9643 ts_tablet_manager.cc:1428] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260926 07:38:12.017073 9642 raft_consensus.cc:374] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:12.017169 9642 raft_consensus.cc:729] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182ccf9aff2f4f1990423e4c8c328bf3, State: Initialized, Role: FOLLOWER
I20260926 07:38:12.016999 9641 consensus_queue.cc:260] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.017939 9642 consensus_queue.cc:260] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.018967 9641 ts_tablet_manager.cc:1428] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 07:38:12.019800 9489 heartbeater.cc:503] Master 127.22.221.126:44917 was elected leader, sending a full tablet report...
I20260926 07:38:12.020691 9642 ts_tablet_manager.cc:1428] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
W20260926 07:38:12.094796 9356 tablet.cc:2369] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:38:12.105748 9490 tablet.cc:2369] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:12.109483 9648 raft_consensus.cc:482] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:12.109779 9648 raft_consensus.cc:504] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.111619 9648 leader_election.cc:288] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182ccf9aff2f4f1990423e4c8c328bf3 (127.22.221.66:43429), 7792756660304b58951426926d5e5879 (127.22.221.67:33549)
I20260926 07:38:12.118945 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.112223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:36123 (local address 127.22.221.67:33549)
0926 07:38:12.112706 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:12.112713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:12.112916 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:38:12.113130 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:12.113136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:12.113216 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:12.113316 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:12.113922 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.114844 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.117092 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.117917 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.118049 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.118145 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:12.118553 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 07:38:12.118680 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260926 07:38:12.119072 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.112099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:60667 (local address 127.22.221.66:43429)
0926 07:38:12.112494 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:12.112502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:12.113134 (+ 632us) server_negotiation.cc:407] Connection header received
0926 07:38:12.113339 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:12.113344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:12.113423 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:12.113557 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:12.114201 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.115116 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.117204 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.118014 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.118175 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.118252 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:12.118649 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 07:38:12.118798 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260926 07:38:12.119454 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.111901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:33549 (local address 127.22.221.65:36123)
0926 07:38:12.112766 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:12.112795 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:12.112972 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:12.113336 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:12.113341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:12.113368 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:12.113767 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:12.113782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.114962 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.114968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:12.116925 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:12.116941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.118155 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.118162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:12.118375 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.118404 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:38:12.118522 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 07:38:12.118659 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":40,"thread_start_us":432,"threads_started":1}
I20260926 07:38:12.120023 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.112070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:43429 (local address 127.22.221.65:60667)
0926 07:38:12.112987 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:12.113001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:12.113179 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:12.113602 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:12.113605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:12.113628 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:12.114015 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:12.114028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.115294 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.115302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:12.117041 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:12.117058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.118272 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.118281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:12.118450 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.118486 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:38:12.119283 (+ 797us) client_negotiation.cc:240] Negotiation successful
0926 07:38:12.119416 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":39,"thread_start_us":571,"threads_started":1}
I20260926 07:38:12.120601 9577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d7e5b181249b7897ce72612000a5e" candidate_uuid: "e12ab23573ed437a9954d39f07da75b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792756660304b58951426926d5e5879" is_pre_election: true
I20260926 07:38:12.121109 9577 raft_consensus.cc:2454] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e12ab23573ed437a9954d39f07da75b4 in term 0.
I20260926 07:38:12.121032 9444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d7e5b181249b7897ce72612000a5e" candidate_uuid: "e12ab23573ed437a9954d39f07da75b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" is_pre_election: true
I20260926 07:38:12.121508 9444 raft_consensus.cc:2454] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e12ab23573ed437a9954d39f07da75b4 in term 0.
I20260926 07:38:12.121837 9243 leader_election.cc:302] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7792756660304b58951426926d5e5879, e12ab23573ed437a9954d39f07da75b4; no voters:
I20260926 07:38:12.122252 9648 raft_consensus.cc:2781] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:38:12.122403 9648 raft_consensus.cc:482] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:38:12.122506 9648 raft_consensus.cc:3037] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:12.124848 9648 raft_consensus.cc:504] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
I20260926 07:38:12.125471 9648 leader_election.cc:288] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [CANDIDATE]: Term 1 election: Requested vote from peers 182ccf9aff2f4f1990423e4c8c328bf3 (127.22.221.66:43429), 7792756660304b58951426926d5e5879 (127.22.221.67:33549)
I20260926 07:38:12.125972 9444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d7e5b181249b7897ce72612000a5e" candidate_uuid: "e12ab23573ed437a9954d39f07da75b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182ccf9aff2f4f1990423e4c8c328bf3"
I20260926 07:38:12.126005 9577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d7e5b181249b7897ce72612000a5e" candidate_uuid: "e12ab23573ed437a9954d39f07da75b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792756660304b58951426926d5e5879"
I20260926 07:38:12.126192 9444 raft_consensus.cc:3037] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:12.126215 9577 raft_consensus.cc:3037] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:12.128522 9444 raft_consensus.cc:2454] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12ab23573ed437a9954d39f07da75b4 in term 1.
I20260926 07:38:12.128543 9577 raft_consensus.cc:2454] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12ab23573ed437a9954d39f07da75b4 in term 1.
I20260926 07:38:12.128999 9244 leader_election.cc:302] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182ccf9aff2f4f1990423e4c8c328bf3, e12ab23573ed437a9954d39f07da75b4; no voters:
I20260926 07:38:12.129318 9648 raft_consensus.cc:2781] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:12.129758 9648 raft_consensus.cc:686] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 1 LEADER]: Becoming Leader. State: Replica: e12ab23573ed437a9954d39f07da75b4, State: Running, Role: LEADER
I20260926 07:38:12.130244 9648 consensus_queue.cc:237] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } }
W20260926 07:38:12.133772 9623 tablet.cc:2369] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:12.134861 9165 catalog_manager.cc:5495] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e12ab23573ed437a9954d39f07da75b4 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "e12ab23573ed437a9954d39f07da75b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e12ab23573ed437a9954d39f07da75b4" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 45729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 } health_report { overall_health: UNKNOWN } } }
I20260926 07:38:12.169521 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.164058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.22.221.65:45729)
0926 07:38:12.164305 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:12.164311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:12.164329 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:38:12.164441 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:12.164445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:12.164536 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:12.164697 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:38:12.165093 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.165824 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.167203 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.167672 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.167805 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.168436 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:38:12.168925 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:38:12.169106 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:12.169246 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:38:12.169295 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:38:12.178119 9577 raft_consensus.cc:1261] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 1 FOLLOWER]: Refusing update from remote peer e12ab23573ed437a9954d39f07da75b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:12.178118 9444 raft_consensus.cc:1261] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 1 FOLLOWER]: Refusing update from remote peer e12ab23573ed437a9954d39f07da75b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:12.179222 9648 consensus_queue.cc:1035] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182ccf9aff2f4f1990423e4c8c328bf3" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 43429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:12.179862 9648 consensus_queue.cc:1035] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7792756660304b58951426926d5e5879" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:12.204932 9165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:12.205231 9165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:12.209112 9165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2ac6e16cba248338306556449ebf125: Sending DeleteTablet for 3 replicas of tablet d93d7e5b181249b7897ce72612000a5e
I20260926 07:38:12.210284 9424 tablet_service.cc:1514] Processing DeleteTablet for tablet d93d7e5b181249b7897ce72612000a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:12 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:57938
I20260926 07:38:12.210515 9557 tablet_service.cc:1514] Processing DeleteTablet for tablet d93d7e5b181249b7897ce72612000a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:12 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:58654
I20260926 07:38:12.210702 9290 tablet_service.cc:1514] Processing DeleteTablet for tablet d93d7e5b181249b7897ce72612000a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:12 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:45718
I20260926 07:38:12.215931 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9227
I20260926 07:38:12.220129 9666 tablet_replica.cc:330] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: stopping tablet replica
I20260926 07:38:12.220597 9666 raft_consensus.cc:2229] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:38:12.221365 9666 raft_consensus.cc:2258] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:12.226362 9666 ts_tablet_manager.cc:1905] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:12.242084 9666 ts_tablet_manager.cc:1918] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:12.242297 9666 log.cc:1196] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/d93d7e5b181249b7897ce72612000a5e
I20260926 07:38:12.242877 9666 ts_tablet_manager.cc:1939] T d93d7e5b181249b7897ce72612000a5e P e12ab23573ed437a9954d39f07da75b4: Deleting consensus metadata
I20260926 07:38:12.245126 9150 catalog_manager.cc:4841] TS e12ab23573ed437a9954d39f07da75b4 (127.22.221.65:45729): tablet d93d7e5b181249b7897ce72612000a5e (table test-table [id=1251078cbac6482aae008d5faba470bf]) successfully deleted
I20260926 07:38:12.255834 9665 tablet_replica.cc:330] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: stopping tablet replica
I20260926 07:38:12.256083 9664 tablet_replica.cc:330] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: stopping tablet replica
I20260926 07:38:12.256429 9665 raft_consensus.cc:2229] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:12.256558 9664 raft_consensus.cc:2229] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 26 07:38:12 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:38:12.257004 9665 raft_consensus.cc:2258] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:12.257107 9664 raft_consensus.cc:2258] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:12.259795 9664 ts_tablet_manager.cc:1905] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:12.266332 9665 ts_tablet_manager.cc:1905] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:12.266822 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.218508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.22.221.65:45729)
0926 07:38:12.218753 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:12.218762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:12.246139 (+ 27377us) server_negotiation.cc:407] Connection header received
0926 07:38:12.246364 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:12.246369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:12.246456 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:12.246540 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:12.248473 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.249710 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.251765 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.252425 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.252512 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.258558 (+ 6046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:12.258577 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:12.258595 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:12.258638 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:12.262891 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:12.264170 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:12.264179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:12.264183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:12.264363 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:12.264817 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:12.264822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:12.264825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:12.265322 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:12.265604 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:12.266220 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 07:38:12.266485 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:38:12.267601 9664 ts_tablet_manager.cc:1918] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:12.267779 9664 log.cc:1196] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/d93d7e5b181249b7897ce72612000a5e
I20260926 07:38:12.268302 9350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:12.268416 9664 ts_tablet_manager.cc:1939] T d93d7e5b181249b7897ce72612000a5e P 182ccf9aff2f4f1990423e4c8c328bf3: Deleting consensus metadata
I20260926 07:38:12.270960 9151 catalog_manager.cc:4841] TS 182ccf9aff2f4f1990423e4c8c328bf3 (127.22.221.66:43429): tablet d93d7e5b181249b7897ce72612000a5e (table test-table [id=1251078cbac6482aae008d5faba470bf]) successfully deleted
I20260926 07:38:12.272287 9665 ts_tablet_manager.cc:1918] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:12.272454 9665 log.cc:1196] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/d93d7e5b181249b7897ce72612000a5e
I20260926 07:38:12.272975 9665 ts_tablet_manager.cc:1939] T d93d7e5b181249b7897ce72612000a5e P 7792756660304b58951426926d5e5879: Deleting consensus metadata
I20260926 07:38:12.274775 9150 catalog_manager.cc:4841] TS 7792756660304b58951426926d5e5879 (127.22.221.67:33549): tablet d93d7e5b181249b7897ce72612000a5e (table test-table [id=1251078cbac6482aae008d5faba470bf]) successfully deleted
I20260926 07:38:12.376494 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9227
I20260926 07:38:12.395366 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9361
Sep 26 07:38:12 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:38:12.410817 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.396259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.22.221.66:43429)
0926 07:38:12.396444 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:12.396448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:12.396466 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:38:12.396620 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:12.396625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:12.396686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:12.396771 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:12.398209 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.398959 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.400451 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.400842 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.400943 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.405134 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:12.405150 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:12.405167 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:12.405204 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:12.408372 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:12.409029 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:12.409036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:12.409038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:12.409153 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:12.409509 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:12.409512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:12.409514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:12.409867 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:12.410123 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:12.410320 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 07:38:12.410477 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:38:12.411609 9484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:12.507267 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9361
I20260926 07:38:12.525741 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9494
Sep 26 07:38:12 dist-test-slave-qlq8 krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:38:12.540887 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:12.526537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.22.221.67:33549)
0926 07:38:12.526708 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:12.526712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:12.526731 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:38:12.526927 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:12.526930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:12.526995 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:12.527079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:12.528481 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.529227 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.530431 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:12.530825 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:12.530974 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:12.535503 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:12.535518 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:12.535534 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:12.535573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:12.538154 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:12.538794 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:12.538800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:12.538803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:12.538962 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:12.539521 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:12.539526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:12.539528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:12.539887 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:12.540151 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:12.540434 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:38:12.540595 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:38:12.541558 9617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:12.632156 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9494
I20260926 07:38:12.648874 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9134
I20260926 07:38:12.650019 9196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:12.763646 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9134
2026-09-26T07:38:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4362 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:12Z Disabled control of system clock
I20260926 07:38:12.841086 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45365
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:38:13.177179 9675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:13.177464 9675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:13.177546 9675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:13.184617 9675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:13.184710 9675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:13.184747 9675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:13.184798 9675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:13.184842 9675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:13.193717 9675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:45365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:13.194999 9675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:13.196732 9675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:13.208333 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:13.208390 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:13.208454 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:13.209115 9675 server_base.cc:1034] running on GCE node
I20260926 07:38:13.209959 9675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:13.211426 9675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:13.212657 9675 hybrid_clock.cc:648] HybridClock initialized: now 1790408293212624 us; error 59 us; skew 500 ppm
I20260926 07:38:13.213012 9675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:13.215517 9675 webserver.cc:460] Webserver started at http://127.22.221.126:44737/ using document root <none> and password file <none>
I20260926 07:38:13.216218 9675 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:13.216352 9675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:13.216668 9675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:13.219528 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "debbf2b4470e4357976f7e33604fcdb1"
format_stamp: "Formatted at 2026-09-26 07:38:13 on dist-test-slave-qlq8"
server_key: "eafc5722dc30172877f0263fb53f959d"
server_key_iv: "6d5e3ded9322856cef00b0f9b9879ae9"
server_key_version: "encryptionkey@0"
I20260926 07:38:13.220340 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "debbf2b4470e4357976f7e33604fcdb1"
format_stamp: "Formatted at 2026-09-26 07:38:13 on dist-test-slave-qlq8"
server_key: "eafc5722dc30172877f0263fb53f959d"
server_key_iv: "6d5e3ded9322856cef00b0f9b9879ae9"
server_key_version: "encryptionkey@0"
I20260926 07:38:13.225309 9675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:38:13.228663 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:13.229988 9675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:38:13.230199 9675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "debbf2b4470e4357976f7e33604fcdb1"
format_stamp: "Formatted at 2026-09-26 07:38:13 on dist-test-slave-qlq8"
server_key: "eafc5722dc30172877f0263fb53f959d"
server_key_iv: "6d5e3ded9322856cef00b0f9b9879ae9"
server_key_version: "encryptionkey@0"
I20260926 07:38:13.230410 9675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:13.248363 9675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:13.249186 9675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:13.249470 9675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:13.268245 9675 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:45365
I20260926 07:38:13.268265 9740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:45365 every 8 connection(s)
I20260926 07:38:13.269634 9675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:13.273484 9741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:13.275586 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9675
I20260926 07:38:13.275781 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:13.276237 23413 external_mini_cluster.cc:1428] Setting key c0d67d08f61a3d025dda0c159f15bfb7
I20260926 07:38:13.285177 9741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1: Bootstrap starting.
I20260926 07:38:13.288594 9741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:13.289670 9741 log.cc:824] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:13.289944 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:13.278208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.22.221.126:45365)
0926 07:38:13.279216 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:13.279287 (+ 71us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:13.279345 (+ 58us) server_negotiation.cc:407] Connection header received
0926 07:38:13.280981 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:13.281003 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:13.281462 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:13.281860 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:13.282484 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:13.283863 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:13.285301 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:13.286027 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:13.286226 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:13.287541 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:13.287550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:13.287564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:13.288096 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:13.288183 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:13.288204 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:13.288339 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:13.288576 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:38:13.288928 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":333,"threads_started":1}
I20260926 07:38:13.292732 9741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1: No bootstrap required, opened a new log
I20260926 07:38:13.299980 9741 raft_consensus.cc:348] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debbf2b4470e4357976f7e33604fcdb1" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45365 } }
I20260926 07:38:13.300352 9741 raft_consensus.cc:374] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:13.300477 9741 raft_consensus.cc:729] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debbf2b4470e4357976f7e33604fcdb1, State: Initialized, Role: FOLLOWER
I20260926 07:38:13.301064 9741 consensus_queue.cc:260] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debbf2b4470e4357976f7e33604fcdb1" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45365 } }
I20260926 07:38:13.301311 9741 raft_consensus.cc:388] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:13.301460 9741 raft_consensus.cc:482] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:13.301676 9741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:13.304136 9741 raft_consensus.cc:504] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debbf2b4470e4357976f7e33604fcdb1" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45365 } }
I20260926 07:38:13.304674 9741 leader_election.cc:302] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debbf2b4470e4357976f7e33604fcdb1; no voters:
I20260926 07:38:13.305260 9741 leader_election.cc:288] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:13.305482 9746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:13.306301 9746 raft_consensus.cc:686] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [term 1 LEADER]: Becoming Leader. State: Replica: debbf2b4470e4357976f7e33604fcdb1, State: Running, Role: LEADER
I20260926 07:38:13.306896 9746 consensus_queue.cc:237] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debbf2b4470e4357976f7e33604fcdb1" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45365 } }
I20260926 07:38:13.307694 9741 sys_catalog.cc:564] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:13.310312 9747 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "debbf2b4470e4357976f7e33604fcdb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debbf2b4470e4357976f7e33604fcdb1" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45365 } } }
I20260926 07:38:13.310724 9747 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:13.310986 9748 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debbf2b4470e4357976f7e33604fcdb1. Latest consensus state: current_term: 1 leader_uuid: "debbf2b4470e4357976f7e33604fcdb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debbf2b4470e4357976f7e33604fcdb1" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45365 } } }
I20260926 07:38:13.311390 9748 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:13.315428 9752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:13.320904 9752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:13.329545 9752 catalog_manager.cc:1347] Generated new cluster ID: 0ba19cfa49dc4c02b8a120184242e797
I20260926 07:38:13.329773 9752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:13.365664 9752 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:13.367617 9752 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:13.400673 9752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1: Generated new TSK 0
I20260926 07:38:13.401666 9752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:13.410392 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45365
--builtin_ntp_servers=127.22.221.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:38:13.749518 9765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:13.749909 9765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:13.750087 9765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:13.757942 9765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:13.758126 9765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:13.758304 9765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:38:13.768208 9765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:13.769624 9765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:13.771562 9765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:13.785614 9771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:13.785645 9773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:13.785689 9770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:13.786538 9765 server_base.cc:1034] running on GCE node
I20260926 07:38:13.787168 9765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:13.788216 9765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:13.789460 9765 hybrid_clock.cc:648] HybridClock initialized: now 1790408293789422 us; error 60 us; skew 500 ppm
I20260926 07:38:13.789791 9765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:13.792480 9765 webserver.cc:460] Webserver started at http://127.22.221.65:41725/ using document root <none> and password file <none>
I20260926 07:38:13.793221 9765 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:13.793365 9765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:13.793694 9765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:13.796854 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "76d87be3469e420096b2cfb23e6a3ad6"
format_stamp: "Formatted at 2026-09-26 07:38:13 on dist-test-slave-qlq8"
server_key: "dbd667d7c28f380bd340f0c9bd8d7103"
server_key_iv: "8813172f2186b5a4153cbd8ef9c54c5d"
server_key_version: "encryptionkey@0"
I20260926 07:38:13.797698 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "76d87be3469e420096b2cfb23e6a3ad6"
format_stamp: "Formatted at 2026-09-26 07:38:13 on dist-test-slave-qlq8"
server_key: "dbd667d7c28f380bd340f0c9bd8d7103"
server_key_iv: "8813172f2186b5a4153cbd8ef9c54c5d"
server_key_version: "encryptionkey@0"
I20260926 07:38:13.802939 9765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:38:13.806632 9779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:13.808095 9765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:38:13.808267 9765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "76d87be3469e420096b2cfb23e6a3ad6"
format_stamp: "Formatted at 2026-09-26 07:38:13 on dist-test-slave-qlq8"
server_key: "dbd667d7c28f380bd340f0c9bd8d7103"
server_key_iv: "8813172f2186b5a4153cbd8ef9c54c5d"
server_key_version: "encryptionkey@0"
I20260926 07:38:13.808492 9765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:13.827276 9765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:13.828145 9765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:13.828470 9765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:13.829455 9765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:13.830943 9765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:13.831080 9765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:13.831202 9765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:13.831328 9765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 07:38:13.865192 9765 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:34755
I20260926 07:38:13.865250 9891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:34755 every 8 connection(s)
I20260926 07:38:13.866593 9765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:38:13.875839 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9765
I20260926 07:38:13.876024 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:38:13.876474 23413 external_mini_cluster.cc:1428] Setting key f1fc4dfde8a51221f96adae397a75b29
I20260926 07:38:13.879509 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45365
--builtin_ntp_servers=127.22.221.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:38:13.879587 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:13.869281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:33161 (local address 127.22.221.126:45365)
0926 07:38:13.869756 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:13.869760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:13.870894 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 07:38:13.872745 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:13.872752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:13.872868 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:13.872994 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:13.874083 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:13.874799 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:13.876556 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:13.877018 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:13.877160 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:13.878705 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:13.878710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:13.878713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:13.878729 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:13.878770 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:13.878782 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:13.878888 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:13.879113 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 07:38:13.879217 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":260,"threads_started":1}
I20260926 07:38:13.880642 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:13.869779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45365 (local address 127.22.221.65:33161)
0926 07:38:13.870704 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:13.870739 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:13.872461 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:13.873237 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:13.873245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:13.873343 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:13.873905 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:13.873922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:13.874959 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:13.874967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:13.876412 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:13.876425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:13.877259 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:13.877268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:13.877574 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:13.878375 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:13.878377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:13.878420 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:13.878507 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:13.878886 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:13.878898 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:13.878909 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:38:13.879104 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 07:38:13.879557 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":272,"threads_started":1}
I20260926 07:38:13.882275 9892 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45365
I20260926 07:38:13.882601 9892 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:13.883463 9892 heartbeater.cc:511] Master 127.22.221.126:45365 requested a full tablet report, sending...
I20260926 07:38:13.885778 9706 ts_manager.cc:194] Registered new tserver with Master: 76d87be3469e420096b2cfb23e6a3ad6 (127.22.221.65:34755)
I20260926 07:38:13.887826 9706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.65:33161
W20260926 07:38:14.214497 9896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:14.214807 9896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:14.214995 9896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:14.222185 9896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:14.222333 9896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:14.222520 9896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:38:14.231740 9896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:14.233119 9896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:14.235249 9896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:14.248787 9904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:14.248872 9902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:14.248831 9901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:14.249869 9896 server_base.cc:1034] running on GCE node
I20260926 07:38:14.250478 9896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:14.251490 9896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:14.252758 9896 hybrid_clock.cc:648] HybridClock initialized: now 1790408294252692 us; error 50 us; skew 500 ppm
I20260926 07:38:14.253127 9896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:14.255656 9896 webserver.cc:460] Webserver started at http://127.22.221.66:40047/ using document root <none> and password file <none>
I20260926 07:38:14.256353 9896 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:14.256493 9896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:14.256809 9896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:14.259773 9896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "2fc6c0c80e4c4f688a2fa88d02020beb"
format_stamp: "Formatted at 2026-09-26 07:38:14 on dist-test-slave-qlq8"
server_key: "2ae489fcd0627484dbe998ee637ce219"
server_key_iv: "0d8faeb6889c596a197d389e7b3068b4"
server_key_version: "encryptionkey@0"
I20260926 07:38:14.260564 9896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "2fc6c0c80e4c4f688a2fa88d02020beb"
format_stamp: "Formatted at 2026-09-26 07:38:14 on dist-test-slave-qlq8"
server_key: "2ae489fcd0627484dbe998ee637ce219"
server_key_iv: "0d8faeb6889c596a197d389e7b3068b4"
server_key_version: "encryptionkey@0"
I20260926 07:38:14.265792 9896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:38:14.269388 9910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:14.270750 9896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:38:14.270925 9896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "2fc6c0c80e4c4f688a2fa88d02020beb"
format_stamp: "Formatted at 2026-09-26 07:38:14 on dist-test-slave-qlq8"
server_key: "2ae489fcd0627484dbe998ee637ce219"
server_key_iv: "0d8faeb6889c596a197d389e7b3068b4"
server_key_version: "encryptionkey@0"
I20260926 07:38:14.271137 9896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:14.288558 9896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:14.289389 9896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:14.289708 9896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:14.290683 9896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:14.292141 9896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:14.292230 9896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:14.292392 9896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:14.292484 9896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:14.325803 9896 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:42787
I20260926 07:38:14.325896 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:42787 every 8 connection(s)
I20260926 07:38:14.327118 9896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:38:14.336580 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 9896
I20260926 07:38:14.336789 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:38:14.337350 23413 external_mini_cluster.cc:1428] Setting key 00cea3d6fa485eaef1c3b2c44956c833
I20260926 07:38:14.340587 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45365
--builtin_ntp_servers=127.22.221.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:38:14.341527 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.329852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:37111 (local address 127.22.221.126:45365)
0926 07:38:14.330043 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.330048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.331534 (+ 1486us) server_negotiation.cc:407] Connection header received
0926 07:38:14.333577 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.333584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.333708 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.333836 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:14.335281 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.336368 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.338464 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.338906 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.339039 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.340618 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:14.340624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:14.340627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:14.340643 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:14.340704 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:14.340722 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:14.340825 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.341102 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.341201 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:38:14.342584 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.330364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45365 (local address 127.22.221.66:37111)
0926 07:38:14.331358 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.331398 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.333274 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.334096 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.334105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.334240 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:14.335060 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.335089 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.336531 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.336538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.338289 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.338309 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.339135 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.339143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.339503 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.340242 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:14.340245 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:14.340310 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:14.340418 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:14.340882 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:14.340891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:14.340897 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.341089 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.341499 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":275,"threads_started":1}
I20260926 07:38:14.344018 10023 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45365
I20260926 07:38:14.344344 10023 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:14.345085 10023 heartbeater.cc:511] Master 127.22.221.126:45365 requested a full tablet report, sending...
I20260926 07:38:14.346549 9706 ts_manager.cc:194] Registered new tserver with Master: 2fc6c0c80e4c4f688a2fa88d02020beb (127.22.221.66:42787)
I20260926 07:38:14.347774 9706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.66:37111
W20260926 07:38:14.671044 10026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:14.671389 10026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:14.671495 10026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:14.678717 10026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:14.678821 10026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:14.678928 10026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:38:14.687846 10026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:14.689070 10026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:14.690738 10026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:14.703934 10032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:14.703881 10034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:14.703848 10031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:14.704658 10026 server_base.cc:1034] running on GCE node
I20260926 07:38:14.705351 10026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:14.706419 10026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:14.707672 10026 hybrid_clock.cc:648] HybridClock initialized: now 1790408294707616 us; error 55 us; skew 500 ppm
I20260926 07:38:14.708004 10026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:14.710270 10026 webserver.cc:460] Webserver started at http://127.22.221.67:34215/ using document root <none> and password file <none>
I20260926 07:38:14.711001 10026 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:14.711133 10026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:14.711586 10026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:14.715607 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "67ce0e16e41549128bc10b938d1f5380"
format_stamp: "Formatted at 2026-09-26 07:38:14 on dist-test-slave-qlq8"
server_key: "71a7f67d9889a6487a7851cfe9b82705"
server_key_iv: "55edab4be2b05b2df27ed0c3ae027f66"
server_key_version: "encryptionkey@0"
I20260926 07:38:14.716861 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "67ce0e16e41549128bc10b938d1f5380"
format_stamp: "Formatted at 2026-09-26 07:38:14 on dist-test-slave-qlq8"
server_key: "71a7f67d9889a6487a7851cfe9b82705"
server_key_iv: "55edab4be2b05b2df27ed0c3ae027f66"
server_key_version: "encryptionkey@0"
I20260926 07:38:14.724751 10026 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 07:38:14.728582 10040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:14.729959 10026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:14.730139 10026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "67ce0e16e41549128bc10b938d1f5380"
format_stamp: "Formatted at 2026-09-26 07:38:14 on dist-test-slave-qlq8"
server_key: "71a7f67d9889a6487a7851cfe9b82705"
server_key_iv: "55edab4be2b05b2df27ed0c3ae027f66"
server_key_version: "encryptionkey@0"
I20260926 07:38:14.730378 10026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:14.753667 10026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:14.754686 10026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:14.755012 10026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:14.756211 10026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:14.757846 10026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:14.757963 10026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:14.758096 10026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:14.758227 10026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:14.791536 10026 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:40861
I20260926 07:38:14.791625 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:40861 every 8 connection(s)
I20260926 07:38:14.792903 10026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:38:14.797619 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10026
I20260926 07:38:14.797808 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:38:14.798444 23413 external_mini_cluster.cc:1428] Setting key 5b8ddc57b2a38c6250527be5c3920d2f
I20260926 07:38:14.806411 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.795660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:33087 (local address 127.22.221.126:45365)
0926 07:38:14.795891 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.795898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.797421 (+ 1523us) server_negotiation.cc:407] Connection header received
0926 07:38:14.799727 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.799733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.799826 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.799939 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:14.801409 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.802158 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.803720 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.804112 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.804205 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.805577 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:14.805583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:14.805586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:14.805603 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:14.805633 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:14.805645 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:14.805746 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.805964 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.806087 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:38:14.807476 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.796316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45365 (local address 127.22.221.67:33087)
0926 07:38:14.797234 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.797281 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.799451 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.800169 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.800177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.800325 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:14.801235 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.801269 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.802292 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.802296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.803600 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.803612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.804296 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.804300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.804491 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.805212 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:14.805215 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:14.805258 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:14.805334 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:14.805746 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:14.805759 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:14.805769 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.805951 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.806339 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
I20260926 07:38:14.808812 10153 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45365
I20260926 07:38:14.809149 10153 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:14.809844 10153 heartbeater.cc:511] Master 127.22.221.126:45365 requested a full tablet report, sending...
I20260926 07:38:14.811278 9706 ts_manager.cc:194] Registered new tserver with Master: 67ce0e16e41549128bc10b938d1f5380 (127.22.221.67:40861)
I20260926 07:38:14.812454 9706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.67:33087
I20260926 07:38:14.813648 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:38:14.823871 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.818182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.22.221.126:45365)
0926 07:38:14.818355 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.818360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.818799 (+ 439us) server_negotiation.cc:407] Connection header received
0926 07:38:14.818930 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.818934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.819004 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.819107 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:14.819768 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.820475 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.821712 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.822109 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.822201 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.823209 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:14.823214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:14.823218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:14.823281 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:14.823324 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:14.823337 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:14.823455 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.823609 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.823653 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:38:14.827674 9706 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:38:14.834697 9706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:38:14.856806 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.846355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:42787 (local address 127.0.0.1:41976)
0926 07:38:14.847339 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.847388 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.847563 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.848688 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.848700 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.848752 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:14.849268 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.849284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.851294 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.851303 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.853888 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.853903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.855807 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.855813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.856036 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.856087 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.856222 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.856408 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":53,"thread_start_us":514,"threads_started":1}
I20260926 07:38:14.856858 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.846527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.22.221.66:42787)
0926 07:38:14.847961 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.847973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.848007 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:38:14.848118 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.848129 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.848411 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.848639 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:14.849559 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.851142 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.854136 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.855593 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.855733 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.856009 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.856283 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.856491 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":1133,"threads_started":1}
I20260926 07:38:14.858106 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.846969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:34755 (local address 127.0.0.1:49418)
0926 07:38:14.850149 (+ 3180us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.850167 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.850276 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.851061 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.851067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.851094 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:14.851720 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.851735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.853085 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.853092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.854949 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.854962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.857311 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.857319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.857486 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.857528 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.857606 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.857728 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3065,"mutex_wait_us":91,"thread_start_us":2572,"threads_started":1}
I20260926 07:38:14.858664 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.846595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:40861 (local address 127.0.0.1:41246)
0926 07:38:14.849898 (+ 3303us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.849925 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.850058 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.850951 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.850955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.850977 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:14.851594 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.851610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.853532 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.853537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.855818 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.855833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.857969 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.857976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.858164 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.858203 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.858277 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.858395 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3126,"thread_start_us":3280,"threads_started":1}
I20260926 07:38:14.859095 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.847533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.22.221.65:34755)
0926 07:38:14.848042 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.848050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.850391 (+ 2341us) server_negotiation.cc:407] Connection header received
0926 07:38:14.850495 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.850502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.850699 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.850889 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:14.851913 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.852891 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.855151 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.856484 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.856620 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.858543 (+ 1923us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.858618 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.858788 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":312,"threads_started":1}
I20260926 07:38:14.859139 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.847262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.22.221.67:40861)
0926 07:38:14.849327 (+ 2065us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.849337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.850186 (+ 849us) server_negotiation.cc:407] Connection header received
0926 07:38:14.850293 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.850300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.850555 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.850738 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:14.851807 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.853361 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.856034 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.857778 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.857918 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.858368 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.858472 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.858633 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":1746,"threads_started":1}
I20260926 07:38:14.861271 9958 tablet_service.cc:1467] Processing CreateTablet for tablet dd3991aad61b4bb790966fd7d13d768c (DEFAULT_TABLE table=test-table [id=a4de4f74e17048ff91e9747f73d395a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:14.862298 10088 tablet_service.cc:1467] Processing CreateTablet for tablet dd3991aad61b4bb790966fd7d13d768c (DEFAULT_TABLE table=test-table [id=a4de4f74e17048ff91e9747f73d395a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:14.862771 9958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3991aad61b4bb790966fd7d13d768c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:14.862552 9827 tablet_service.cc:1467] Processing CreateTablet for tablet dd3991aad61b4bb790966fd7d13d768c (DEFAULT_TABLE table=test-table [id=a4de4f74e17048ff91e9747f73d395a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:14.863694 10088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3991aad61b4bb790966fd7d13d768c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:14.863842 9827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3991aad61b4bb790966fd7d13d768c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:14.878309 10171 tablet_bootstrap.cc:492] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Bootstrap starting.
I20260926 07:38:14.881582 10171 tablet_bootstrap.cc:654] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:14.882622 10171 log.cc:824] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:14.883661 10173 tablet_bootstrap.cc:492] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Bootstrap starting.
I20260926 07:38:14.884608 10172 tablet_bootstrap.cc:492] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Bootstrap starting.
I20260926 07:38:14.888337 10173 tablet_bootstrap.cc:654] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:14.888746 10172 tablet_bootstrap.cc:654] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:14.889698 10172 log.cc:824] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:14.889672 10173 log.cc:824] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:14.890123 10171 tablet_bootstrap.cc:492] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: No bootstrap required, opened a new log
I20260926 07:38:14.890502 10171 ts_tablet_manager.cc:1397] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 07:38:14.890964 9892 heartbeater.cc:503] Master 127.22.221.126:45365 was elected leader, sending a full tablet report...
I20260926 07:38:14.893013 10172 tablet_bootstrap.cc:492] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: No bootstrap required, opened a new log
I20260926 07:38:14.893280 10172 ts_tablet_manager.cc:1397] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:38:14.899034 10173 tablet_bootstrap.cc:492] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: No bootstrap required, opened a new log
I20260926 07:38:14.899369 10173 ts_tablet_manager.cc:1397] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260926 07:38:14.901783 10171 raft_consensus.cc:348] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.902187 10171 raft_consensus.cc:374] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:14.902287 10171 raft_consensus.cc:729] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ce0e16e41549128bc10b938d1f5380, State: Initialized, Role: FOLLOWER
I20260926 07:38:14.902966 10171 consensus_queue.cc:260] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.904129 10153 heartbeater.cc:503] Master 127.22.221.126:45365 was elected leader, sending a full tablet report...
I20260926 07:38:14.904291 10172 raft_consensus.cc:348] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.904737 10172 raft_consensus.cc:374] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:14.904825 10172 raft_consensus.cc:729] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc6c0c80e4c4f688a2fa88d02020beb, State: Initialized, Role: FOLLOWER
I20260926 07:38:14.904910 10171 ts_tablet_manager.cc:1428] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260926 07:38:14.905409 10172 consensus_queue.cc:260] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.907315 10023 heartbeater.cc:503] Master 127.22.221.126:45365 was elected leader, sending a full tablet report...
I20260926 07:38:14.907859 10172 ts_tablet_manager.cc:1428] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260926 07:38:14.911238 10173 raft_consensus.cc:348] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.911760 10173 raft_consensus.cc:374] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:14.911938 10173 raft_consensus.cc:729] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d87be3469e420096b2cfb23e6a3ad6, State: Initialized, Role: FOLLOWER
I20260926 07:38:14.912598 10173 consensus_queue.cc:260] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.914198 10173 ts_tablet_manager.cc:1428] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260926 07:38:14.957511 10177 raft_consensus.cc:482] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:14.957801 10177 raft_consensus.cc:504] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.959548 10177 leader_election.cc:288] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc6c0c80e4c4f688a2fa88d02020beb (127.22.221.66:42787), 76d87be3469e420096b2cfb23e6a3ad6 (127.22.221.65:34755)
I20260926 07:38:14.965241 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.960155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:58335 (local address 127.22.221.65:34755)
0926 07:38:14.960341 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.960347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.960368 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:38:14.960450 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.960453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.960526 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.960654 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:14.961117 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.961804 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.963467 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.964315 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.964484 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.964553 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.964843 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.964966 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:38:14.965242 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.959825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:34755 (local address 127.22.221.67:58335)
0926 07:38:14.960029 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.960051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.960197 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.960713 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.960719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.960745 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:14.960937 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.960945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.961944 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.961949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.963313 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.963325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.964478 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.964484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.964671 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.964694 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.964806 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.964944 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:38:14.966061 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.960123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:53723 (local address 127.22.221.66:42787)
0926 07:38:14.960367 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:14.960373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:14.960923 (+ 550us) server_negotiation.cc:407] Connection header received
0926 07:38:14.961161 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:14.961165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:14.961222 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:14.961322 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:14.961803 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.962492 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.964380 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.965140 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.965315 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.965382 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:14.965698 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:38:14.965824 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 07:38:14.966138 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:14.959820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:42787 (local address 127.22.221.67:53723)
0926 07:38:14.960807 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:14.960821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:14.961005 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:14.961338 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:14.961342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:14.961364 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:14.961657 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.961665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.962642 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.962647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.964204 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:14.964215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:14.965312 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:14.965318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:14.965503 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:14.965548 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:38:14.965668 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:38:14.965788 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":227,"thread_start_us":446,"threads_started":1}
I20260926 07:38:14.966595 9847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3991aad61b4bb790966fd7d13d768c" candidate_uuid: "67ce0e16e41549128bc10b938d1f5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d87be3469e420096b2cfb23e6a3ad6" is_pre_election: true
I20260926 07:38:14.967123 9847 raft_consensus.cc:2454] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67ce0e16e41549128bc10b938d1f5380 in term 0.
I20260926 07:38:14.967088 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3991aad61b4bb790966fd7d13d768c" candidate_uuid: "67ce0e16e41549128bc10b938d1f5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" is_pre_election: true
I20260926 07:38:14.967603 9978 raft_consensus.cc:2454] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67ce0e16e41549128bc10b938d1f5380 in term 0.
I20260926 07:38:14.967875 10043 leader_election.cc:302] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67ce0e16e41549128bc10b938d1f5380, 76d87be3469e420096b2cfb23e6a3ad6; no voters:
I20260926 07:38:14.968274 10177 raft_consensus.cc:2781] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:38:14.968416 10177 raft_consensus.cc:482] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:38:14.968516 10177 raft_consensus.cc:3037] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:14.970715 10177 raft_consensus.cc:504] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.971447 10177 leader_election.cc:288] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [CANDIDATE]: Term 1 election: Requested vote from peers 2fc6c0c80e4c4f688a2fa88d02020beb (127.22.221.66:42787), 76d87be3469e420096b2cfb23e6a3ad6 (127.22.221.65:34755)
I20260926 07:38:14.971863 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3991aad61b4bb790966fd7d13d768c" candidate_uuid: "67ce0e16e41549128bc10b938d1f5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb"
I20260926 07:38:14.972012 9847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3991aad61b4bb790966fd7d13d768c" candidate_uuid: "67ce0e16e41549128bc10b938d1f5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d87be3469e420096b2cfb23e6a3ad6"
I20260926 07:38:14.972079 9978 raft_consensus.cc:3037] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:14.972234 9847 raft_consensus.cc:3037] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:14.975421 9847 raft_consensus.cc:2454] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67ce0e16e41549128bc10b938d1f5380 in term 1.
I20260926 07:38:14.975428 9978 raft_consensus.cc:2454] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67ce0e16e41549128bc10b938d1f5380 in term 1.
I20260926 07:38:14.975844 10042 leader_election.cc:302] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc6c0c80e4c4f688a2fa88d02020beb, 67ce0e16e41549128bc10b938d1f5380; no voters:
I20260926 07:38:14.976161 10177 raft_consensus.cc:2781] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:14.976611 10177 raft_consensus.cc:686] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 1 LEADER]: Becoming Leader. State: Replica: 67ce0e16e41549128bc10b938d1f5380, State: Running, Role: LEADER
I20260926 07:38:14.977187 10177 consensus_queue.cc:237] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } }
I20260926 07:38:14.981704 9706 catalog_manager.cc:5495] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67ce0e16e41549128bc10b938d1f5380 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "67ce0e16e41549128bc10b938d1f5380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67ce0e16e41549128bc10b938d1f5380" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 } health_report { overall_health: UNKNOWN } } }
I20260926 07:38:15.024410 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:15.018564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.22.221.67:40861)
0926 07:38:15.018792 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:15.018799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:15.018911 (+ 112us) server_negotiation.cc:407] Connection header received
0926 07:38:15.019168 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:15.019172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:15.019323 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:15.019479 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:38:15.019881 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.020603 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.022024 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.022435 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.022571 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:15.023302 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:38:15.023824 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:38:15.024017 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:15.024131 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 07:38:15.024177 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:38:15.032670 9978 raft_consensus.cc:1261] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 1 FOLLOWER]: Refusing update from remote peer 67ce0e16e41549128bc10b938d1f5380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:15.032670 9847 raft_consensus.cc:1261] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 67ce0e16e41549128bc10b938d1f5380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:15.033653 10181 consensus_queue.cc:1035] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d87be3469e420096b2cfb23e6a3ad6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:15.034090 10177 consensus_queue.cc:1035] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc6c0c80e4c4f688a2fa88d02020beb" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 42787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:38:15.044559 10154 tablet.cc:2369] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:15.046518 10184 mvcc.cc:205] Tried to move back new op lower bound from 7333512376443416576 to 7333512376230612992. Current Snapshot: MvccSnapshot[applied={T|T < 7333512376443416576}]
I20260926 07:38:15.064850 9706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:15.065143 9706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:15.068919 9706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P debbf2b4470e4357976f7e33604fcdb1: Sending DeleteTablet for 3 replicas of tablet dd3991aad61b4bb790966fd7d13d768c
I20260926 07:38:15.070093 9827 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3991aad61b4bb790966fd7d13d768c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:15 UTC) from {username='slave'} at 127.0.0.1:49418
I20260926 07:38:15.070091 10088 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3991aad61b4bb790966fd7d13d768c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:15 UTC) from {username='slave'} at 127.0.0.1:41246
I20260926 07:38:15.070891 9958 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3991aad61b4bb790966fd7d13d768c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:15 UTC) from {username='slave'} at 127.0.0.1:41976
I20260926 07:38:15.071115 10194 tablet_replica.cc:330] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: stopping tablet replica
I20260926 07:38:15.071624 10194 raft_consensus.cc:2229] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:15.072217 10194 raft_consensus.cc:2258] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:15.072925 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9765
I20260926 07:38:15.073509 10195 tablet_replica.cc:330] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: stopping tablet replica
I20260926 07:38:15.073999 10195 raft_consensus.cc:2229] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:38:15.074725 10195 raft_consensus.cc:2258] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:15.076910 10195 ts_tablet_manager.cc:1905] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:15.076910 10194 ts_tablet_manager.cc:1905] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:15.083982 10194 ts_tablet_manager.cc:1918] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:15.084149 10196 tablet_replica.cc:330] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: stopping tablet replica
I20260926 07:38:15.084206 10194 log.cc:1196] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/dd3991aad61b4bb790966fd7d13d768c
I20260926 07:38:15.084815 10194 ts_tablet_manager.cc:1939] T dd3991aad61b4bb790966fd7d13d768c P 76d87be3469e420096b2cfb23e6a3ad6: Deleting consensus metadata
I20260926 07:38:15.084723 10196 raft_consensus.cc:2229] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:15.085383 10196 raft_consensus.cc:2258] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:15.085443 10195 ts_tablet_manager.cc:1918] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:15.085731 10195 log.cc:1196] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/dd3991aad61b4bb790966fd7d13d768c
I20260926 07:38:15.086302 10195 ts_tablet_manager.cc:1939] T dd3991aad61b4bb790966fd7d13d768c P 67ce0e16e41549128bc10b938d1f5380: Deleting consensus metadata
I20260926 07:38:15.086827 9692 catalog_manager.cc:4841] TS 76d87be3469e420096b2cfb23e6a3ad6 (127.22.221.65:34755): tablet dd3991aad61b4bb790966fd7d13d768c (table test-table [id=a4de4f74e17048ff91e9747f73d395a2]) successfully deleted
I20260926 07:38:15.088460 10196 ts_tablet_manager.cc:1905] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:15.088984 9693 catalog_manager.cc:4841] TS 67ce0e16e41549128bc10b938d1f5380 (127.22.221.67:40861): tablet dd3991aad61b4bb790966fd7d13d768c (table test-table [id=a4de4f74e17048ff91e9747f73d395a2]) successfully deleted
I20260926 07:38:15.094571 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:15.075149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.22.221.65:34755)
0926 07:38:15.075354 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:15.075363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:15.086080 (+ 10717us) server_negotiation.cc:407] Connection header received
0926 07:38:15.086600 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:15.086605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:15.086729 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:15.086809 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:15.087489 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.088607 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.090672 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.091372 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.091518 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:15.092860 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:15.092872 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:15.092893 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:15.093687 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:15.093755 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:15.093781 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:15.093914 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:15.094085 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:38:15.094205 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:38:15.094695 10196 ts_tablet_manager.cc:1918] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:15.094897 10196 log.cc:1196] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/dd3991aad61b4bb790966fd7d13d768c
I20260926 07:38:15.095397 9887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:15.095443 10196 ts_tablet_manager.cc:1939] T dd3991aad61b4bb790966fd7d13d768c P 2fc6c0c80e4c4f688a2fa88d02020beb: Deleting consensus metadata
I20260926 07:38:15.097043 9691 catalog_manager.cc:4841] TS 2fc6c0c80e4c4f688a2fa88d02020beb (127.22.221.66:42787): tablet dd3991aad61b4bb790966fd7d13d768c (table test-table [id=a4de4f74e17048ff91e9747f73d395a2]) successfully deleted
I20260926 07:38:15.190146 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9765
I20260926 07:38:15.214900 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9896
I20260926 07:38:15.229250 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:15.215870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.22.221.66:42787)
0926 07:38:15.216137 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:15.216143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:15.216165 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:38:15.216235 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:15.216240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:15.216339 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:15.216448 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:15.216893 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.217647 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.218940 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.219425 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.219566 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:15.227714 (+ 8148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:15.227728 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:15.227747 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:15.228429 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:15.228486 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:15.228508 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:15.228645 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:15.228792 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:38:15.228909 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 07:38:15.229972 10018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:15.336220 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9896
I20260926 07:38:15.354815 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10026
I20260926 07:38:15.362314 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:15.355752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.22.221.67:40861)
0926 07:38:15.355951 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:15.355957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:15.355977 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:38:15.356180 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:15.356185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:15.356281 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:15.356401 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:15.356846 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.357686 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.358985 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:15.359558 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:15.359702 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:15.360846 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:15.360855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:15.360872 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:15.361518 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:15.361573 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:15.361592 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:15.361715 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:15.361862 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:38:15.361972 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:38:15.363288 10148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:15.488211 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10026
I20260926 07:38:15.507668 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 9675
I20260926 07:38:15.508895 9736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:15.620069 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 9675
2026-09-26T07:38:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2855 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:15 dist-test-slave-qlq8 krb5kdc[10204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:15 dist-test-slave-qlq8 krb5kdc[10204](info): set up 2 sockets
Sep 26 07:38:15 dist-test-slave-qlq8 krb5kdc[10204](info): commencing operation
krb5kdc: starting...
W20260926 07:38:16.751689 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:16 dist-test-slave-qlq8 krb5kdc[10204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:17.011080 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37523
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:17.344658 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:17.345006 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:17.345151 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:17.353327 10217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:17.353436 10217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:17.353489 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:17.353549 10217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:17.353592 10217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:17.363317 10217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:17.364655 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:17.365954 10217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:17.378134 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:17.378153 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:17.378093 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:17.379007 10217 server_base.cc:1034] running on GCE node
I20260926 07:38:17.379925 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:17.381318 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:17.382553 10217 hybrid_clock.cc:648] HybridClock initialized: now 1790408297382504 us; error 41 us; skew 500 ppm
I20260926 07:38:17.382916 10217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:17 dist-test-slave-qlq8 krb5kdc[10204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408297, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:17.385990 10217 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:17.387481 10217 webserver.cc:460] Webserver started at http://127.22.221.126:37439/ using document root <none> and password file <none>
I20260926 07:38:17.388104 10217 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:17.388200 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:17.388437 10217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:17.391360 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "415b6bd18cae432cbf447484b840a871"
format_stamp: "Formatted at 2026-09-26 07:38:17 on dist-test-slave-qlq8"
server_key: "54e1f95471f4ee9623301b2c1057cd28"
server_key_iv: "be1c7db3a0c54cde3ec9b6a05acd17ec"
server_key_version: "encryptionkey@0"
I20260926 07:38:17.392053 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "415b6bd18cae432cbf447484b840a871"
format_stamp: "Formatted at 2026-09-26 07:38:17 on dist-test-slave-qlq8"
server_key: "54e1f95471f4ee9623301b2c1057cd28"
server_key_iv: "be1c7db3a0c54cde3ec9b6a05acd17ec"
server_key_version: "encryptionkey@0"
I20260926 07:38:17.396929 10217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:38:17.400166 10232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:17.401420 10217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:38:17.401562 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "415b6bd18cae432cbf447484b840a871"
format_stamp: "Formatted at 2026-09-26 07:38:17 on dist-test-slave-qlq8"
server_key: "54e1f95471f4ee9623301b2c1057cd28"
server_key_iv: "be1c7db3a0c54cde3ec9b6a05acd17ec"
server_key_version: "encryptionkey@0"
I20260926 07:38:17.401713 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:17.414083 10217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:17.417852 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:17.418083 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:17.436833 10217 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37523
I20260926 07:38:17.436843 10283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37523 every 8 connection(s)
I20260926 07:38:17.438253 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:17.442037 10284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:17.445429 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10217
I20260926 07:38:17.445616 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:17.446188 23413 external_mini_cluster.cc:1428] Setting key 7ecbd37e5bdec4bc091a31063a7de702
I20260926 07:38:17.453871 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871: Bootstrap starting.
I20260926 07:38:17.457031 10284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:17.458387 10284 log.cc:824] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:38:17 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:17.461313 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871: No bootstrap required, opened a new log
I20260926 07:38:17.470433 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:17.448068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.22.221.126:37523)
0926 07:38:17.448802 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:17.448813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:17.448853 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:38:17.450482 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:17.450503 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:17.450906 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:17.451322 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:17.452852 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:17.454199 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:17.455675 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:17.456345 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:17.456549 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:17.461367 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:17.461374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:17.461387 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:17.461453 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:17.465435 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:17.466225 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:17.466236 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:17.466250 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:17.466498 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:17.466883 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:17.466888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:17.466891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:17.467669 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:17.468013 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:17.468399 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 07:38:17.468927 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":324,"threads_started":1}
I20260926 07:38:17.472193 10284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415b6bd18cae432cbf447484b840a871" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37523 } }
I20260926 07:38:17.472585 10284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:17.472712 10284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415b6bd18cae432cbf447484b840a871, State: Initialized, Role: FOLLOWER
I20260926 07:38:17.473405 10284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415b6bd18cae432cbf447484b840a871" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37523 } }
I20260926 07:38:17.473723 10284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:17.473860 10284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:17.474011 10284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:17.476361 10284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415b6bd18cae432cbf447484b840a871" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37523 } }
I20260926 07:38:17.476804 10284 leader_election.cc:302] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415b6bd18cae432cbf447484b840a871; no voters:
I20260926 07:38:17.477479 10284 leader_election.cc:288] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:17.477643 10289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:17.478915 10289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [term 1 LEADER]: Becoming Leader. State: Replica: 415b6bd18cae432cbf447484b840a871, State: Running, Role: LEADER
I20260926 07:38:17.479527 10289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415b6bd18cae432cbf447484b840a871" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37523 } }
I20260926 07:38:17.479888 10284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:17.482568 10291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415b6bd18cae432cbf447484b840a871. Latest consensus state: current_term: 1 leader_uuid: "415b6bd18cae432cbf447484b840a871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415b6bd18cae432cbf447484b840a871" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37523 } } }
I20260926 07:38:17.482986 10291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:17.482534 10290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415b6bd18cae432cbf447484b840a871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415b6bd18cae432cbf447484b840a871" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37523 } } }
I20260926 07:38:17.483460 10290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:17.486245 10296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:17.492630 10296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:17.502652 10296 catalog_manager.cc:1347] Generated new cluster ID: 54cfbec1e2944230b33063eef981d503
I20260926 07:38:17.502779 10296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:17.537192 10296 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:17.538535 10296 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:17.555514 10296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871: Generated new TSK 0
I20260926 07:38:17.556654 10296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
Entry for principal kudu/127.22.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab.
I20260926 07:38:17.618984 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37523
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:17.957687 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:17.958037 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:17.958206 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:17.965745 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:17.965924 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:17.966102 10310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:38:17.976220 10310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.65.keytab
--principal=kudu/127.22.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:17.977626 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:17.979054 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:17.992892 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:17.992868 10318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:17.992870 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:17.999101 10310 server_base.cc:1034] running on GCE node
I20260926 07:38:17.999950 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:18.001021 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:18.002269 10310 hybrid_clock.cc:648] HybridClock initialized: now 1790408298002200 us; error 75 us; skew 500 ppm
I20260926 07:38:18.002612 10310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:18 dist-test-slave-qlq8 krb5kdc[10204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408298, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:18.006179 10310 init.cc:377] Logged in from keytab as kudu/127.22.221.65@KRBTEST.COM (short username kudu)
I20260926 07:38:18.007891 10310 webserver.cc:460] Webserver started at http://127.22.221.65:34557/ using document root <none> and password file <none>
I20260926 07:38:18.008631 10310 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:18.008777 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:18.009097 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:18.012079 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "6345b20c7c974e77a3e13f6f3de1b2d6"
format_stamp: "Formatted at 2026-09-26 07:38:18 on dist-test-slave-qlq8"
server_key: "b40dd8e5fc58104e4b07f492171cef2b"
server_key_iv: "8e275931d3cdd864e48948bd6e7c1ffe"
server_key_version: "encryptionkey@0"
I20260926 07:38:18.012837 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "6345b20c7c974e77a3e13f6f3de1b2d6"
format_stamp: "Formatted at 2026-09-26 07:38:18 on dist-test-slave-qlq8"
server_key: "b40dd8e5fc58104e4b07f492171cef2b"
server_key_iv: "8e275931d3cdd864e48948bd6e7c1ffe"
server_key_version: "encryptionkey@0"
I20260926 07:38:18.018057 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:18.021464 10325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:18.022819 10310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:38:18.023005 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "6345b20c7c974e77a3e13f6f3de1b2d6"
format_stamp: "Formatted at 2026-09-26 07:38:18 on dist-test-slave-qlq8"
server_key: "b40dd8e5fc58104e4b07f492171cef2b"
server_key_iv: "8e275931d3cdd864e48948bd6e7c1ffe"
server_key_version: "encryptionkey@0"
I20260926 07:38:18.023205 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:18.043464 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:18.048036 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:18.048365 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:18.049391 10310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:18.050815 10310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:18.050925 10310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:18.051090 10310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:18.051163 10310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:18.085413 10310 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:36129
I20260926 07:38:18.085455 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:36129 every 8 connection(s)
I20260926 07:38:18.086738 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:38:18.087285 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10310
I20260926 07:38:18.087497 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:38:18.088129 23413 external_mini_cluster.cc:1428] Setting key 9e27f2cfd6723a64612ddeb83d36c501
Sep 26 07:38:18 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408298, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.65@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:18.111480 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:18.089730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:52623 (local address 127.22.221.126:37523)
0926 07:38:18.090366 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:18.090372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:18.091706 (+ 1334us) server_negotiation.cc:407] Connection header received
0926 07:38:18.094707 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:18.094711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:18.094779 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:18.094878 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:18.097079 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.097790 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.100392 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.100988 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.101143 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:18.104752 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:18.104761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:18.104768 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:18.104814 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:18.107892 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:18.108939 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:18.108944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:18.108947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:18.109074 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:18.109611 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:18.109615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:18.109617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:18.109879 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:18.110000 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:18.110963 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 07:38:18.111143 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":237,"threads_started":1}
I20260926 07:38:18.113039 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:18.090600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37523 (local address 127.22.221.65:52623)
0926 07:38:18.091512 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:18.091577 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:18.094382 (+ 2805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:18.095174 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:18.095188 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:18.096037 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:18.096873 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:18.096902 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.097966 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.097974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:18.100228 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:18.100246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.101210 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.101215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:18.101414 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:18.102147 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:18.102175 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:18.104477 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:18.108050 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:18.108063 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:18.108084 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:18.108801 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:18.109192 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:18.109200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:18.109206 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:18.109489 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:18.110012 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:18.110023 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:18.110519 (+ 496us) client_negotiation.cc:769] Sending connection context
0926 07:38:18.110870 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 07:38:18.111500 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":355,"threads_started":1}
I20260926 07:38:18.114820 10438 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37523
I20260926 07:38:18.115145 10438 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:18.115926 10438 heartbeater.cc:511] Master 127.22.221.126:37523 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.221.66@KRBTEST.COM; defaulting to no policy
I20260926 07:38:18.118183 10249 ts_manager.cc:194] Registered new tserver with Master: 6345b20c7c974e77a3e13f6f3de1b2d6 (127.22.221.65:36129)
I20260926 07:38:18.119910 10249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.65@KRBTEST.COM'} at 127.22.221.65:52623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
Entry for principal kudu/127.22.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab.
I20260926 07:38:18.150916 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37523
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:18.508442 10444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:18.508721 10444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:18.508890 10444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:18.516168 10444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:18.516273 10444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:18.516484 10444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:38:18.526301 10444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.66.keytab
--principal=kudu/127.22.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:18.527694 10444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:18.528972 10444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:18.542771 10449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:18.542773 10450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:18.543058 10452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:18.543749 10444 server_base.cc:1034] running on GCE node
I20260926 07:38:18.544430 10444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:18.545475 10444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:18.546693 10444 hybrid_clock.cc:648] HybridClock initialized: now 1790408298546639 us; error 45 us; skew 500 ppm
I20260926 07:38:18.547061 10444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:18 dist-test-slave-qlq8 krb5kdc[10204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408298, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:18.550310 10444 init.cc:377] Logged in from keytab as kudu/127.22.221.66@KRBTEST.COM (short username kudu)
I20260926 07:38:18.551880 10444 webserver.cc:460] Webserver started at http://127.22.221.66:35417/ using document root <none> and password file <none>
I20260926 07:38:18.552580 10444 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:18.552731 10444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:18.553051 10444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:18.556113 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "3369086030b348ef926e452462cbd8b9"
format_stamp: "Formatted at 2026-09-26 07:38:18 on dist-test-slave-qlq8"
server_key: "396e7d90ef3be02b11ef8aba9058157c"
server_key_iv: "3740ebd0c75ac0190facc1a4803b51b6"
server_key_version: "encryptionkey@0"
I20260926 07:38:18.556927 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "3369086030b348ef926e452462cbd8b9"
format_stamp: "Formatted at 2026-09-26 07:38:18 on dist-test-slave-qlq8"
server_key: "396e7d90ef3be02b11ef8aba9058157c"
server_key_iv: "3740ebd0c75ac0190facc1a4803b51b6"
server_key_version: "encryptionkey@0"
I20260926 07:38:18.562523 10444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:18.566272 10459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:18.567824 10444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:18.568020 10444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "3369086030b348ef926e452462cbd8b9"
format_stamp: "Formatted at 2026-09-26 07:38:18 on dist-test-slave-qlq8"
server_key: "396e7d90ef3be02b11ef8aba9058157c"
server_key_iv: "3740ebd0c75ac0190facc1a4803b51b6"
server_key_version: "encryptionkey@0"
I20260926 07:38:18.568251 10444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:18.590685 10444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:18.595033 10444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:18.595415 10444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:18.596443 10444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:18.597775 10444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:18.597883 10444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:18.598035 10444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:18.598109 10444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:18.632033 10444 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:44255
I20260926 07:38:18.632090 10571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:44255 every 8 connection(s)
I20260926 07:38:18.633374 10444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:38:18.637272 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10444
I20260926 07:38:18.637496 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:38:18.638010 23413 external_mini_cluster.cc:1428] Setting key 134457bac511ca013bc5a090ba723f56
Sep 26 07:38:18 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408298, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.66@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:18.654140 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:18.636121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:60195 (local address 127.22.221.126:37523)
0926 07:38:18.636590 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:18.636596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:18.637729 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 07:38:18.639723 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:18.639729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:18.639820 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:18.639937 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:18.641469 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.642248 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.643954 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.644395 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.644528 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:18.648388 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:18.648393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:18.648396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:18.648430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:18.651066 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:18.651832 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:18.651837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:18.651839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:18.651971 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:18.652399 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:18.652405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:18.652407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:18.652655 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:18.652809 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:18.653514 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 07:38:18.653739 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
I20260926 07:38:18.655105 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:18.636720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37523 (local address 127.22.221.66:60195)
0926 07:38:18.637542 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:18.637593 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:18.639464 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:18.640182 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:18.640190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:18.640720 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:18.641291 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:18.641316 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.642414 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.642420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:18.643792 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:18.643812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:18.644630 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:18.644637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:18.644874 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:18.645586 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:18.645608 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:18.648133 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:18.651212 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:18.651221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:18.651258 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:18.651695 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:18.652082 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:18.652087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:18.652090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:18.652274 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:18.652782 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:18.652789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:18.653130 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 07:38:18.653429 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 07:38:18.653976 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":355,"threads_started":1}
I20260926 07:38:18.656672 10572 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37523
I20260926 07:38:18.657060 10572 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:18.657984 10572 heartbeater.cc:511] Master 127.22.221.126:37523 requested a full tablet report, sending...
I20260926 07:38:18.659705 10249 ts_manager.cc:194] Registered new tserver with Master: 3369086030b348ef926e452462cbd8b9 (127.22.221.66:44255)
I20260926 07:38:18.660595 10249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.66@KRBTEST.COM'} at 127.22.221.66:60195
WARNING: no policy specified for kudu/127.22.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
Entry for principal kudu/127.22.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab.
I20260926 07:38:18.698678 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37523
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:19.030429 10578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:19.030814 10578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:19.030993 10578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:19.038465 10578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:19.038640 10578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:19.038842 10578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:38:19.048899 10578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.67.keytab
--principal=kudu/127.22.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:37523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:19.050339 10578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:19.051820 10578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:19.065153 10586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:19.065218 10583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:19.065272 10584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:19.066116 10578 server_base.cc:1034] running on GCE node
I20260926 07:38:19.066841 10578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:19.067974 10578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:19.069216 10578 hybrid_clock.cc:648] HybridClock initialized: now 1790408299069177 us; error 49 us; skew 500 ppm
I20260926 07:38:19.069555 10578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:19 dist-test-slave-qlq8 krb5kdc[10204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408299, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:19.073329 10578 init.cc:377] Logged in from keytab as kudu/127.22.221.67@KRBTEST.COM (short username kudu)
I20260926 07:38:19.074949 10578 webserver.cc:460] Webserver started at http://127.22.221.67:41513/ using document root <none> and password file <none>
I20260926 07:38:19.075800 10578 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:19.075970 10578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:19.076325 10578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:19.079713 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "4f052483486a49c38cc9ba2ff85485d1"
format_stamp: "Formatted at 2026-09-26 07:38:19 on dist-test-slave-qlq8"
server_key: "2e9705f10c3b10f0babdb9bd43c1da19"
server_key_iv: "2354ebb83997a7cd6ad2586d8df7b3a6"
server_key_version: "encryptionkey@0"
I20260926 07:38:19.080662 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "4f052483486a49c38cc9ba2ff85485d1"
format_stamp: "Formatted at 2026-09-26 07:38:19 on dist-test-slave-qlq8"
server_key: "2e9705f10c3b10f0babdb9bd43c1da19"
server_key_iv: "2354ebb83997a7cd6ad2586d8df7b3a6"
server_key_version: "encryptionkey@0"
I20260926 07:38:19.086388 10578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:19.090410 10593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:19.091850 10578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:19.092077 10578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "4f052483486a49c38cc9ba2ff85485d1"
format_stamp: "Formatted at 2026-09-26 07:38:19 on dist-test-slave-qlq8"
server_key: "2e9705f10c3b10f0babdb9bd43c1da19"
server_key_iv: "2354ebb83997a7cd6ad2586d8df7b3a6"
server_key_version: "encryptionkey@0"
I20260926 07:38:19.092299 10578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:19.115159 10578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:19.120339 10578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:19.120680 10578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:19.121783 10578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:19.123057 10438 heartbeater.cc:503] Master 127.22.221.126:37523 was elected leader, sending a full tablet report...
I20260926 07:38:19.123598 10578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:19.123744 10578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:19.123930 10578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:19.124006 10578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:19.161379 10578 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:36429
I20260926 07:38:19.161420 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:36429 every 8 connection(s)
I20260926 07:38:19.162829 10578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:38:19.164165 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10578
I20260926 07:38:19.164351 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:38:19.164911 23413 external_mini_cluster.cc:1428] Setting key 04bd2fdb26113ada9097939769ebf033
Sep 26 07:38:19 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408299, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.67@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:19.184873 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.166095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:49419 (local address 127.22.221.126:37523)
0926 07:38:19.166587 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.166594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.167832 (+ 1238us) server_negotiation.cc:407] Connection header received
0926 07:38:19.169807 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.169811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.169895 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.169995 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:19.171692 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.172606 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.174234 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.174670 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.174775 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.178652 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:19.178659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:19.178664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:19.178705 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:19.182073 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.182770 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.182774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.182777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.182894 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.183326 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.183329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.183332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.183618 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:19.183743 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.184386 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.184569 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260926 07:38:19.185801 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.166702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:37523 (local address 127.22.221.67:49419)
0926 07:38:19.167647 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.167709 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.169534 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.170243 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.170252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.170854 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:19.171512 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.171531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.172773 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.172777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.174097 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.174109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.174864 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.174870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.175325 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.176093 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:38:19.176121 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:19.178411 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:19.182202 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:19.182209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:19.182222 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:19.182654 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:19.182994 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:38:19.182997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:38:19.183000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:38:19.183176 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:38:19.183740 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:19.183747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:19.184059 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.184285 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.184663 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":341,"threads_started":1}
I20260926 07:38:19.187192 10706 heartbeater.cc:347] Connected to a master server at 127.22.221.126:37523
I20260926 07:38:19.187537 10706 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:19.188319 10706 heartbeater.cc:511] Master 127.22.221.126:37523 requested a full tablet report, sending...
I20260926 07:38:19.189803 10249 ts_manager.cc:194] Registered new tserver with Master: 4f052483486a49c38cc9ba2ff85485d1 (127.22.221.67:36429)
I20260926 07:38:19.190616 10249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.221.67@KRBTEST.COM'} at 127.22.221.67:49419
I20260926 07:38:19.193152 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:38:19.211805 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.197484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.22.221.126:37523)
0926 07:38:19.197701 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.197711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.198104 (+ 393us) server_negotiation.cc:407] Connection header received
0926 07:38:19.198334 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.198338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.198426 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.198548 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:19.200233 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.201020 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.202482 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.202938 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.203089 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.205962 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:19.205967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:19.205971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:19.206010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:19.208809 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.209581 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.209595 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.209598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.209726 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.210131 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.210137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.210144 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.210514 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:19.210794 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.211172 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.211472 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:38:19.216048 10249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:38:19.223719 10249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:38:19.261408 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.237731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.22.221.67:36429)
0926 07:38:19.250053 (+ 12322us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.250067 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.252234 (+ 2167us) server_negotiation.cc:407] Connection header received
0926 07:38:19.252951 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.252958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.253176 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.253341 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.254390 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.255914 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.258498 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.259991 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.260071 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.260339 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.260870 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.261045 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12076,"thread_start_us":12328,"threads_started":1}
I20260926 07:38:19.261466 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.237159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:36429 (local address 127.0.0.1:40946)
0926 07:38:19.252099 (+ 14940us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.252121 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.252758 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.253688 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.253695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.253720 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.254199 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.254211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.256109 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.256113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.258277 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.258296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.260403 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.260409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.260602 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.260659 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.260801 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.261013 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14782,"mutex_wait_us":35,"thread_start_us":14615,"threads_started":1}
I20260926 07:38:19.262246 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.236859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:36129 (local address 127.0.0.1:58044)
0926 07:38:19.251015 (+ 14156us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.251071 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.251347 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.253427 (+ 2080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.253439 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.253492 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.254038 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.254053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.256078 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.256086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.258095 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.258110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.261394 (+ 3284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.261402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.261583 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.261619 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.261711 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.261863 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13803,"mutex_wait_us":24,"thread_start_us":13622,"threads_started":1}
I20260926 07:38:19.262476 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.237042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.22.221.65:36129)
0926 07:38:19.252767 (+ 15725us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.252775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.252803 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:38:19.252906 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.252914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.253172 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.254008 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.254314 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.255935 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.258329 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.260897 (+ 2568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.261019 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.261248 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.261950 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.262118 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15564,"thread_start_us":15409,"threads_started":1}
I20260926 07:38:19.263396 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.252452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.22.221.66:44255)
0926 07:38:19.252986 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.252994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.253891 (+ 897us) server_negotiation.cc:407] Connection header received
0926 07:38:19.254216 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.254223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.254484 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.254657 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.255857 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.257456 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.260269 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.261926 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.262081 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.262250 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.262900 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.263061 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":296,"threads_started":1}
I20260926 07:38:19.263475 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.253038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:44255 (local address 127.0.0.1:50210)
0926 07:38:19.253698 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.253717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.253971 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.254932 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.254936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.254961 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.255668 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.255680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.257617 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.257625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.260124 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.260140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.262513 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.262520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.262685 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.262728 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.262848 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.263014 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"spinlock_wait_cycles":4992,"thread_start_us":333,"threads_started":1}
I20260926 07:38:19.265776 10641 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfbc2c24d0f49f99c653d83c348d851 (DEFAULT_TABLE table=test-table [id=5d740b7fca274f159920323e7cdd97a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:19.267062 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfbc2c24d0f49f99c653d83c348d851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:19.267175 10507 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfbc2c24d0f49f99c653d83c348d851 (DEFAULT_TABLE table=test-table [id=5d740b7fca274f159920323e7cdd97a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:19.267175 10373 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfbc2c24d0f49f99c653d83c348d851 (DEFAULT_TABLE table=test-table [id=5d740b7fca274f159920323e7cdd97a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:19.268627 10373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfbc2c24d0f49f99c653d83c348d851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:19.268687 10507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfbc2c24d0f49f99c653d83c348d851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:19.278524 10725 tablet_bootstrap.cc:492] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Bootstrap starting.
I20260926 07:38:19.282222 10725 tablet_bootstrap.cc:654] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:19.283283 10726 tablet_bootstrap.cc:492] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Bootstrap starting.
I20260926 07:38:19.283555 10725 log.cc:824] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:19.284148 10727 tablet_bootstrap.cc:492] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Bootstrap starting.
I20260926 07:38:19.287881 10726 tablet_bootstrap.cc:654] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:19.287882 10727 tablet_bootstrap.cc:654] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:19.289073 10725 tablet_bootstrap.cc:492] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: No bootstrap required, opened a new log
I20260926 07:38:19.289161 10727 log.cc:824] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:19.289161 10726 log.cc:824] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:19.289399 10725 ts_tablet_manager.cc:1397] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260926 07:38:19.292501 10726 tablet_bootstrap.cc:492] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: No bootstrap required, opened a new log
I20260926 07:38:19.292784 10726 ts_tablet_manager.cc:1397] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 07:38:19.292831 10727 tablet_bootstrap.cc:492] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: No bootstrap required, opened a new log
I20260926 07:38:19.293082 10727 ts_tablet_manager.cc:1397] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:38:19.300441 10726 raft_consensus.cc:348] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.300467 10725 raft_consensus.cc:348] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.300882 10725 raft_consensus.cc:374] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:19.300913 10726 raft_consensus.cc:374] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:19.301020 10725 raft_consensus.cc:729] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f052483486a49c38cc9ba2ff85485d1, State: Initialized, Role: FOLLOWER
I20260926 07:38:19.301046 10726 raft_consensus.cc:729] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6345b20c7c974e77a3e13f6f3de1b2d6, State: Initialized, Role: FOLLOWER
I20260926 07:38:19.301666 10725 consensus_queue.cc:260] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.301740 10726 consensus_queue.cc:260] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.303563 10706 heartbeater.cc:503] Master 127.22.221.126:37523 was elected leader, sending a full tablet report...
I20260926 07:38:19.303702 10727 raft_consensus.cc:348] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.304128 10727 raft_consensus.cc:374] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:19.304203 10725 ts_tablet_manager.cc:1428] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260926 07:38:19.304226 10727 raft_consensus.cc:729] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3369086030b348ef926e452462cbd8b9, State: Initialized, Role: FOLLOWER
I20260926 07:38:19.304971 10727 consensus_queue.cc:260] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.306005 10726 ts_tablet_manager.cc:1428] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260926 07:38:19.307050 10572 heartbeater.cc:503] Master 127.22.221.126:37523 was elected leader, sending a full tablet report...
I20260926 07:38:19.307667 10727 ts_tablet_manager.cc:1428] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
W20260926 07:38:19.339535 10439 tablet.cc:2369] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:38:19.385380 10573 tablet.cc:2369] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:38:19.415038 10707 tablet.cc:2369] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:19.545778 10732 raft_consensus.cc:482] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:19.546099 10732 raft_consensus.cc:504] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.547813 10732 leader_election.cc:288] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f052483486a49c38cc9ba2ff85485d1 (127.22.221.67:36429), 3369086030b348ef926e452462cbd8b9 (127.22.221.66:44255)
I20260926 07:38:19.554368 10733 raft_consensus.cc:482] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:19.554945 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:44161 (local address 127.22.221.67:36429)
0926 07:38:19.548403 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.548408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.548879 (+ 471us) server_negotiation.cc:407] Connection header received
0926 07:38:19.549112 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.549118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.549200 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.549299 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.550096 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.551102 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.553093 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.554027 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.554148 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.554198 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.554619 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.554730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:38:19.554854 10733 raft_consensus.cc:504] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.555547 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.548881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:44255 (local address 127.22.221.65:58307)
0926 07:38:19.549568 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.549585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.549697 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.550078 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.550083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.550106 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.550500 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.550513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.551617 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.551621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.553273 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.553289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.554593 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.554602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.554781 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.554821 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.554923 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.555079 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":44,"thread_start_us":346,"threads_started":1}
I20260926 07:38:19.555680 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.549226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:58307 (local address 127.22.221.66:44255)
0926 07:38:19.549500 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.549506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.549673 (+ 167us) server_negotiation.cc:407] Connection header received
0926 07:38:19.549834 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.549839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.549922 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.550060 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.550652 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.551487 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.553456 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.554340 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.554530 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.554617 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.554967 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.555105 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:38:19.555898 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.548077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:36429 (local address 127.22.221.65:44161)
0926 07:38:19.548767 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.548789 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.548932 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.549479 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.549483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.549514 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.549926 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.549939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.551261 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.551267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.552944 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.552955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.554232 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.554239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.554414 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.554456 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.555188 (+ 732us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.555355 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":50,"thread_start_us":367,"threads_started":1}
I20260926 07:38:19.556720 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfbc2c24d0f49f99c653d83c348d851" candidate_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3369086030b348ef926e452462cbd8b9" is_pre_election: true
I20260926 07:38:19.556934 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfbc2c24d0f49f99c653d83c348d851" candidate_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f052483486a49c38cc9ba2ff85485d1" is_pre_election: true
I20260926 07:38:19.557152 10527 raft_consensus.cc:2454] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6345b20c7c974e77a3e13f6f3de1b2d6 in term 0.
I20260926 07:38:19.557449 10661 raft_consensus.cc:2454] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6345b20c7c974e77a3e13f6f3de1b2d6 in term 0.
I20260926 07:38:19.557937 10327 leader_election.cc:302] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3369086030b348ef926e452462cbd8b9, 6345b20c7c974e77a3e13f6f3de1b2d6; no voters:
I20260926 07:38:19.558346 10732 raft_consensus.cc:2781] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:38:19.558568 10732 raft_consensus.cc:482] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:38:19.558738 10732 raft_consensus.cc:3037] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:19.559876 10733 leader_election.cc:288] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6345b20c7c974e77a3e13f6f3de1b2d6 (127.22.221.65:36129), 4f052483486a49c38cc9ba2ff85485d1 (127.22.221.67:36429)
I20260926 07:38:19.562222 10732 raft_consensus.cc:504] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.563249 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfbc2c24d0f49f99c653d83c348d851" candidate_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f052483486a49c38cc9ba2ff85485d1"
I20260926 07:38:19.563489 10661 raft_consensus.cc:3037] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:19.564206 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfbc2c24d0f49f99c653d83c348d851" candidate_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3369086030b348ef926e452462cbd8b9"
I20260926 07:38:19.564401 10527 raft_consensus.cc:3037] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:19.565912 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.558406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:39717 (local address 127.22.221.65:36129)
0926 07:38:19.558917 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.558922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.558959 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:38:19.559010 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.559018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.559095 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.559185 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.559960 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.561043 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.563615 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.564826 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.564977 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.565059 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.565559 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.565682 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
I20260926 07:38:19.565958 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.559483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:39867 (local address 127.22.221.67:36429)
0926 07:38:19.559645 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.559652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.559670 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:38:19.559896 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.559902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.559993 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.560058 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:19.560663 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.561448 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.563316 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.564656 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.564825 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.564917 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.565541 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.565674 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:38:19.566708 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.558086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:36129 (local address 127.22.221.66:39717)
0926 07:38:19.558738 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.558757 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.558872 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.559363 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.559367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.559409 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.559779 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.559791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.561222 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.561226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.563396 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.563412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.565198 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.565204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.565371 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.565412 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.566144 (+ 732us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.566348 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":375,"threads_started":1}
I20260926 07:38:19.566780 10732 leader_election.cc:288] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [CANDIDATE]: Term 1 election: Requested vote from peers 4f052483486a49c38cc9ba2ff85485d1 (127.22.221.67:36429), 3369086030b348ef926e452462cbd8b9 (127.22.221.66:44255)
I20260926 07:38:19.566903 10661 raft_consensus.cc:2454] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6345b20c7c974e77a3e13f6f3de1b2d6 in term 1.
I20260926 07:38:19.567384 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.559005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:36429 (local address 127.22.221.66:39867)
0926 07:38:19.559600 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:19.559617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:19.559741 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:19.560228 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:19.560231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:19.560249 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:19.560518 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.560525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.561599 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.561607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.563144 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:19.563156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.565093 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.565100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:19.565315 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.565354 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:38:19.566157 (+ 803us) client_negotiation.cc:240] Negotiation successful
0926 07:38:19.566587 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":284,"thread_start_us":659,"threads_started":1}
I20260926 07:38:19.567543 10327 leader_election.cc:302] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f052483486a49c38cc9ba2ff85485d1, 6345b20c7c974e77a3e13f6f3de1b2d6; no voters:
I20260926 07:38:19.567709 10527 raft_consensus.cc:2454] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6345b20c7c974e77a3e13f6f3de1b2d6 in term 1.
I20260926 07:38:19.567936 10732 raft_consensus.cc:2781] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:19.567855 10393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfbc2c24d0f49f99c653d83c348d851" candidate_uuid: "3369086030b348ef926e452462cbd8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" is_pre_election: true
I20260926 07:38:19.568449 10732 raft_consensus.cc:686] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6345b20c7c974e77a3e13f6f3de1b2d6, State: Running, Role: LEADER
I20260926 07:38:19.568555 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfbc2c24d0f49f99c653d83c348d851" candidate_uuid: "3369086030b348ef926e452462cbd8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f052483486a49c38cc9ba2ff85485d1" is_pre_election: true
I20260926 07:38:19.568856 10661 raft_consensus.cc:2379] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3369086030b348ef926e452462cbd8b9 in current term 1: Already voted for candidate 6345b20c7c974e77a3e13f6f3de1b2d6 in this term.
I20260926 07:38:19.569270 10732 consensus_queue.cc:237] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } }
I20260926 07:38:19.573444 10463 leader_election.cc:302] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3369086030b348ef926e452462cbd8b9; no voters: 4f052483486a49c38cc9ba2ff85485d1, 6345b20c7c974e77a3e13f6f3de1b2d6
I20260926 07:38:19.573925 10733 raft_consensus.cc:2726] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 07:38:19.574790 10247 catalog_manager.cc:5495] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6345b20c7c974e77a3e13f6f3de1b2d6 (127.22.221.65). New cstate: current_term: 1 leader_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6345b20c7c974e77a3e13f6f3de1b2d6" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 36129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 } health_report { overall_health: UNKNOWN } } }
I20260926 07:38:19.628875 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.623202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.22.221.65:36129)
0926 07:38:19.623414 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.623420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.623463 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:38:19.623747 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.623751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.623831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.623986 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:38:19.624379 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.625173 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.626615 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.627118 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.627289 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.627780 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:38:19.628273 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:38:19.628442 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.628496 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.628577 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:38:19.636868 10661 raft_consensus.cc:1261] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 1 FOLLOWER]: Refusing update from remote peer 6345b20c7c974e77a3e13f6f3de1b2d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:19.636971 10527 raft_consensus.cc:1261] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 1 FOLLOWER]: Refusing update from remote peer 6345b20c7c974e77a3e13f6f3de1b2d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:19.637744 10732 consensus_queue.cc:1035] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f052483486a49c38cc9ba2ff85485d1" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 36429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:19.638060 10738 consensus_queue.cc:1035] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3369086030b348ef926e452462cbd8b9" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:19.650156 10741 mvcc.cc:205] Tried to move back new op lower bound from 7333512395303010304 to 7333512395039608832. Current Snapshot: MvccSnapshot[applied={T|T < 7333512395303010304}]
I20260926 07:38:19.665983 10248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:19.666318 10248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:19.670461 10248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 415b6bd18cae432cbf447484b840a871: Sending DeleteTablet for 3 replicas of tablet 1bfbc2c24d0f49f99c653d83c348d851
I20260926 07:38:19.671731 10373 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bfbc2c24d0f49f99c653d83c348d851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:19 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:58044
I20260926 07:38:19.671882 10641 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bfbc2c24d0f49f99c653d83c348d851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:19 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:40946
I20260926 07:38:19.671882 10507 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bfbc2c24d0f49f99c653d83c348d851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:19 UTC) from {username='kudu', principal='kudu/127.22.221.126@KRBTEST.COM'} at 127.0.0.1:50210
I20260926 07:38:19.674947 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10310
I20260926 07:38:19.681808 10751 tablet_replica.cc:330] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: stopping tablet replica
I20260926 07:38:19.682382 10751 raft_consensus.cc:2229] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:38:19.683204 10751 raft_consensus.cc:2258] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:19.685546 10751 ts_tablet_manager.cc:1905] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:19.693746 10751 ts_tablet_manager.cc:1918] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:19.694052 10751 log.cc:1196] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/1bfbc2c24d0f49f99c653d83c348d851
I20260926 07:38:19.694756 10751 ts_tablet_manager.cc:1939] T 1bfbc2c24d0f49f99c653d83c348d851 P 6345b20c7c974e77a3e13f6f3de1b2d6: Deleting consensus metadata
I20260926 07:38:19.697152 10236 catalog_manager.cc:4841] TS 6345b20c7c974e77a3e13f6f3de1b2d6 (127.22.221.65:36129): tablet 1bfbc2c24d0f49f99c653d83c348d851 (table test-table [id=5d740b7fca274f159920323e7cdd97a3]) successfully deleted
I20260926 07:38:19.700971 10753 tablet_replica.cc:330] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: stopping tablet replica
I20260926 07:38:19.701406 10752 tablet_replica.cc:330] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: stopping tablet replica
I20260926 07:38:19.701730 10753 raft_consensus.cc:2229] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:19.701866 10752 raft_consensus.cc:2229] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:19.702310 10753 raft_consensus.cc:2258] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:19.702399 10752 raft_consensus.cc:2258] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:19.704739 10752 ts_tablet_manager.cc:1905] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:19.704773 10753 ts_tablet_manager.cc:1905] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:38:19 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.65@KRBTEST.COM
I20260926 07:38:19.712621 10752 ts_tablet_manager.cc:1918] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:19.712826 10753 ts_tablet_manager.cc:1918] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:19.712910 10752 log.cc:1196] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/1bfbc2c24d0f49f99c653d83c348d851
I20260926 07:38:19.713073 10753 log.cc:1196] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/1bfbc2c24d0f49f99c653d83c348d851
I20260926 07:38:19.713570 10752 ts_tablet_manager.cc:1939] T 1bfbc2c24d0f49f99c653d83c348d851 P 3369086030b348ef926e452462cbd8b9: Deleting consensus metadata
I20260926 07:38:19.713932 10753 ts_tablet_manager.cc:1939] T 1bfbc2c24d0f49f99c653d83c348d851 P 4f052483486a49c38cc9ba2ff85485d1: Deleting consensus metadata
I20260926 07:38:19.715611 10234 catalog_manager.cc:4841] TS 3369086030b348ef926e452462cbd8b9 (127.22.221.66:44255): tablet 1bfbc2c24d0f49f99c653d83c348d851 (table test-table [id=5d740b7fca274f159920323e7cdd97a3]) successfully deleted
I20260926 07:38:19.715919 10234 catalog_manager.cc:4841] TS 4f052483486a49c38cc9ba2ff85485d1 (127.22.221.67:36429): tablet 1bfbc2c24d0f49f99c653d83c348d851 (table test-table [id=5d740b7fca274f159920323e7cdd97a3]) successfully deleted
I20260926 07:38:19.718712 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.676797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.22.221.65:36129)
0926 07:38:19.677086 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.677096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.698226 (+ 21130us) server_negotiation.cc:407] Connection header received
0926 07:38:19.698532 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.698537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.698634 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.698714 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:19.701253 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.702406 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.704563 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.705204 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.705324 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.711003 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:19.711024 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:19.711046 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:19.711105 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:19.715325 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.716244 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.716254 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.716258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.716434 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.716869 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.716874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.716878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.717421 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:19.717786 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.717989 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.718278 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:38:19.719522 10433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:19.826743 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10310
I20260926 07:38:19.845937 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10444
Sep 26 07:38:19 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.66@KRBTEST.COM
I20260926 07:38:19.861616 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.846742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.22.221.66:44255)
0926 07:38:19.846924 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.846930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.847053 (+ 123us) server_negotiation.cc:407] Connection header received
0926 07:38:19.847311 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.847316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.847415 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.847534 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:19.848859 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.849570 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.850969 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.851513 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.851651 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.855994 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:19.856006 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:19.856022 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:19.856057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:19.858739 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.859530 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.859537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.859539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.859686 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.860068 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.860074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.860077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.860434 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:19.860733 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.861029 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.861273 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:38:19.862219 10567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:19.954711 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10444
I20260926 07:38:19.972585 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10578
Sep 26 07:38:19 dist-test-slave-qlq8 krb5kdc[10204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.67@KRBTEST.COM
I20260926 07:38:19.987610 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:19.973369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.22.221.67:36429)
0926 07:38:19.973539 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:19.973543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:19.973561 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:38:19.973687 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:19.973690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:19.973772 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:19.973871 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:19.975201 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.975926 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.977232 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:19.977649 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:19.977744 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:19.982344 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:19.982359 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:19.982375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:19.982408 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:19.985203 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.985829 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.985837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.985839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.985962 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:19.986294 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:19.986297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:19.986299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:19.986635 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:19.986892 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:19.987141 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 07:38:19.987374 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:38:19.988325 10701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:20.077976 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10578
I20260926 07:38:20.094733 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10217
I20260926 07:38:20.096011 10279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:20.207157 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10217
2026-09-26T07:38:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4587 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:20Z Disabled control of system clock
I20260926 07:38:20.284354 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45885
--webserver_interface=127.22.221.126
--webserver_port=0
--builtin_ntp_servers=127.22.221.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:45885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:38:20.621599 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:20.621888 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:20.622028 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:20.629392 10762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:20.629505 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:20.629575 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:20.629664 10762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:20.629727 10762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:20.639088 10762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:45885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:45885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:20.640416 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:20.642279 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:20.654691 10768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:20.654712 10767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:20.654755 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:20.655660 10762 server_base.cc:1034] running on GCE node
I20260926 07:38:20.656499 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:20.657971 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:20.659202 10762 hybrid_clock.cc:648] HybridClock initialized: now 1790408300659150 us; error 66 us; skew 500 ppm
I20260926 07:38:20.659597 10762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:20.667398 10762 webserver.cc:460] Webserver started at http://127.22.221.126:45591/ using document root <none> and password file <none>
I20260926 07:38:20.668243 10762 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:20.668380 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:20.668717 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:20.671891 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "325137589cd14cf79fe2b062aef3f65f"
format_stamp: "Formatted at 2026-09-26 07:38:20 on dist-test-slave-qlq8"
server_key: "06e27e9db53fd85e79a517fd4d544d93"
server_key_iv: "01707104d7fb8e2aede0ba9566e398e5"
server_key_version: "encryptionkey@0"
I20260926 07:38:20.672744 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "325137589cd14cf79fe2b062aef3f65f"
format_stamp: "Formatted at 2026-09-26 07:38:20 on dist-test-slave-qlq8"
server_key: "06e27e9db53fd85e79a517fd4d544d93"
server_key_iv: "01707104d7fb8e2aede0ba9566e398e5"
server_key_version: "encryptionkey@0"
I20260926 07:38:20.678248 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:20.681885 10776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:20.683519 10762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:20.683713 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "325137589cd14cf79fe2b062aef3f65f"
format_stamp: "Formatted at 2026-09-26 07:38:20 on dist-test-slave-qlq8"
server_key: "06e27e9db53fd85e79a517fd4d544d93"
server_key_iv: "01707104d7fb8e2aede0ba9566e398e5"
server_key_version: "encryptionkey@0"
I20260926 07:38:20.683954 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:20.713846 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:20.714759 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:20.715065 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:20.733810 10762 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:45885
I20260926 07:38:20.733829 10827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:45885 every 8 connection(s)
I20260926 07:38:20.735190 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:20.738924 10828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:20.739579 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10762
I20260926 07:38:20.739758 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:20.740267 23413 external_mini_cluster.cc:1428] Setting key 2cc854b79f15f274538f3dd7677e67b9
I20260926 07:38:20.753415 10828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f: Bootstrap starting.
I20260926 07:38:20.754715 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:20.742516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.22.221.126:45885)
0926 07:38:20.743619 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:20.743630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:20.743672 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:38:20.745300 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:20.745322 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:20.745820 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:20.746203 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:20.746878 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:20.748234 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:20.750202 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:20.750958 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:20.751100 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:20.752419 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:20.752427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:20.752444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:20.753001 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:20.753080 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:20.753101 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:20.753192 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:20.753459 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 07:38:20.753778 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":412,"threads_started":1}
I20260926 07:38:20.756882 10828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:20.758083 10828 log.cc:824] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:20.761410 10828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f: No bootstrap required, opened a new log
I20260926 07:38:20.769271 10828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325137589cd14cf79fe2b062aef3f65f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45885 } }
I20260926 07:38:20.769685 10828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:20.769802 10828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325137589cd14cf79fe2b062aef3f65f, State: Initialized, Role: FOLLOWER
I20260926 07:38:20.770452 10828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325137589cd14cf79fe2b062aef3f65f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45885 } }
I20260926 07:38:20.770680 10828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:20.770879 10828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:20.771072 10828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:20.773473 10828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325137589cd14cf79fe2b062aef3f65f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45885 } }
I20260926 07:38:20.774109 10828 leader_election.cc:302] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325137589cd14cf79fe2b062aef3f65f; no voters:
I20260926 07:38:20.774823 10828 leader_election.cc:288] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:20.775007 10833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:20.775836 10833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [term 1 LEADER]: Becoming Leader. State: Replica: 325137589cd14cf79fe2b062aef3f65f, State: Running, Role: LEADER
I20260926 07:38:20.776459 10833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325137589cd14cf79fe2b062aef3f65f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45885 } }
I20260926 07:38:20.777230 10828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:20.779577 10834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "325137589cd14cf79fe2b062aef3f65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325137589cd14cf79fe2b062aef3f65f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45885 } } }
I20260926 07:38:20.780092 10834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:20.780438 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325137589cd14cf79fe2b062aef3f65f. Latest consensus state: current_term: 1 leader_uuid: "325137589cd14cf79fe2b062aef3f65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325137589cd14cf79fe2b062aef3f65f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 45885 } } }
I20260926 07:38:20.780843 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:20.782087 10841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:20.787721 10841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:20.796164 10841 catalog_manager.cc:1347] Generated new cluster ID: 8e5a23de52664c8cbd541047d6e088b1
I20260926 07:38:20.796295 10841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:20.807436 10841 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:20.808765 10841 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:20.824267 10841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f: Generated new TSK 0
I20260926 07:38:20.825198 10841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:20.843611 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.65:0
--local_ip_for_outbound_sockets=127.22.221.65
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45885
--builtin_ntp_servers=127.22.221.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:38:21.176424 10852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:21.176741 10852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:21.176918 10852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:21.184829 10852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:21.184930 10852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:21.185204 10852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.65
I20260926 07:38:21.194752 10852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.221.65
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.65
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:21.196139 10852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:21.197945 10852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:21.211555 10857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:21.211640 10860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:21.211678 10858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:21.212554 10852 server_base.cc:1034] running on GCE node
I20260926 07:38:21.213128 10852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:21.214084 10852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:21.215337 10852 hybrid_clock.cc:648] HybridClock initialized: now 1790408301215272 us; error 44 us; skew 500 ppm
I20260926 07:38:21.215667 10852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:21.217898 10852 webserver.cc:460] Webserver started at http://127.22.221.65:38621/ using document root <none> and password file <none>
I20260926 07:38:21.218549 10852 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:21.218633 10852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:21.218895 10852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:21.221824 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data/instance:
uuid: "bb0ac665e57b459da291084a1fcbdd26"
format_stamp: "Formatted at 2026-09-26 07:38:21 on dist-test-slave-qlq8"
server_key: "dd6e022dd9cb0acb9b17b9b6e4336408"
server_key_iv: "a46b5e9f63f35c07535cd2902a7754cd"
server_key_version: "encryptionkey@0"
I20260926 07:38:21.222576 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance:
uuid: "bb0ac665e57b459da291084a1fcbdd26"
format_stamp: "Formatted at 2026-09-26 07:38:21 on dist-test-slave-qlq8"
server_key: "dd6e022dd9cb0acb9b17b9b6e4336408"
server_key_iv: "a46b5e9f63f35c07535cd2902a7754cd"
server_key_version: "encryptionkey@0"
I20260926 07:38:21.227767 10852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:38:21.231299 10866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:21.232709 10852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:21.232908 10852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal
uuid: "bb0ac665e57b459da291084a1fcbdd26"
format_stamp: "Formatted at 2026-09-26 07:38:21 on dist-test-slave-qlq8"
server_key: "dd6e022dd9cb0acb9b17b9b6e4336408"
server_key_iv: "a46b5e9f63f35c07535cd2902a7754cd"
server_key_version: "encryptionkey@0"
I20260926 07:38:21.233134 10852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:21.261377 10852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:21.262301 10852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:21.262660 10852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:21.263707 10852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:21.265193 10852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:21.265326 10852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:21.265458 10852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:21.265545 10852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:21.299475 10852 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.65:44359
I20260926 07:38:21.299558 10978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.65:44359 every 8 connection(s)
I20260926 07:38:21.300769 10852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/data/info.pb
I20260926 07:38:21.309476 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10852
I20260926 07:38:21.309666 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal/instance
I20260926 07:38:21.310209 23413 external_mini_cluster.cc:1428] Setting key f7442807f3e120e1b13d939cce194e22
I20260926 07:38:21.313342 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.66:0
--local_ip_for_outbound_sockets=127.22.221.66
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45885
--builtin_ntp_servers=127.22.221.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:38:21.313875 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:21.303359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.65:53971 (local address 127.22.221.126:45885)
0926 07:38:21.303848 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:21.303852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:21.304861 (+ 1009us) server_negotiation.cc:407] Connection header received
0926 07:38:21.306651 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:21.306657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:21.306753 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:21.306874 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:21.308013 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.308757 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.311044 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.311525 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.311663 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:21.313047 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:21.313051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:21.313055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:21.313071 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:21.313108 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:21.313120 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:21.313190 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:21.313522 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 07:38:21.313618 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
I20260926 07:38:21.315408 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:21.303842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45885 (local address 127.22.221.65:53971)
0926 07:38:21.304701 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:21.304737 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:21.306374 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:21.307113 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:21.307121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:21.307210 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:21.307834 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:21.307855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.308906 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.308918 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:21.310898 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:21.310911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.311746 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.311756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:21.312002 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:21.312692 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:21.312695 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:21.312758 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:21.312856 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:21.313226 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:21.313239 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:21.313247 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:38:21.313471 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:38:21.313884 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":254,"threads_started":1}
I20260926 07:38:21.316922 10979 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45885
I20260926 07:38:21.317209 10979 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:21.317937 10979 heartbeater.cc:511] Master 127.22.221.126:45885 requested a full tablet report, sending...
I20260926 07:38:21.320003 10793 ts_manager.cc:194] Registered new tserver with Master: bb0ac665e57b459da291084a1fcbdd26 (127.22.221.65:44359)
I20260926 07:38:21.321656 10793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.65:53971
W20260926 07:38:21.640904 10983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:21.641166 10983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:21.641250 10983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:21.648166 10983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:21.648340 10983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:21.648532 10983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.66
I20260926 07:38:21.657464 10983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.221.66
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.66
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:21.658743 10983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:21.660558 10983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:21.673540 10991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:21.673615 10988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:21.673611 10989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:21.674835 10983 server_base.cc:1034] running on GCE node
I20260926 07:38:21.675578 10983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:21.676561 10983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:21.677769 10983 hybrid_clock.cc:648] HybridClock initialized: now 1790408301677724 us; error 51 us; skew 500 ppm
I20260926 07:38:21.678143 10983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:21.680581 10983 webserver.cc:460] Webserver started at http://127.22.221.66:37259/ using document root <none> and password file <none>
I20260926 07:38:21.681272 10983 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:21.681383 10983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:21.681705 10983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:21.684669 10983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data/instance:
uuid: "ecbda375ec044c4ebc908d3b0a0c23f8"
format_stamp: "Formatted at 2026-09-26 07:38:21 on dist-test-slave-qlq8"
server_key: "fe3f3a7c028435b9f4cfd03f9210bdbe"
server_key_iv: "41c0ecff61dbc41f77bff67efc527973"
server_key_version: "encryptionkey@0"
I20260926 07:38:21.685570 10983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance:
uuid: "ecbda375ec044c4ebc908d3b0a0c23f8"
format_stamp: "Formatted at 2026-09-26 07:38:21 on dist-test-slave-qlq8"
server_key: "fe3f3a7c028435b9f4cfd03f9210bdbe"
server_key_iv: "41c0ecff61dbc41f77bff67efc527973"
server_key_version: "encryptionkey@0"
I20260926 07:38:21.690754 10983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:38:21.694198 10997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:21.695617 10983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:38:21.695787 10983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal
uuid: "ecbda375ec044c4ebc908d3b0a0c23f8"
format_stamp: "Formatted at 2026-09-26 07:38:21 on dist-test-slave-qlq8"
server_key: "fe3f3a7c028435b9f4cfd03f9210bdbe"
server_key_iv: "41c0ecff61dbc41f77bff67efc527973"
server_key_version: "encryptionkey@0"
I20260926 07:38:21.695987 10983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:21.717828 10983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:21.718660 10983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:21.718979 10983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:21.720058 10983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:21.721493 10983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:21.721611 10983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:21.721742 10983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:21.721856 10983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:21.759492 10983 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.66:41673
I20260926 07:38:21.759577 11109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.66:41673 every 8 connection(s)
I20260926 07:38:21.760772 10983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/data/info.pb
I20260926 07:38:21.770648 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 10983
I20260926 07:38:21.770884 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal/instance
I20260926 07:38:21.771474 23413 external_mini_cluster.cc:1428] Setting key d415105628ae1f93dee5fa15b83a9794
I20260926 07:38:21.773767 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:21.763349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.66:37451 (local address 127.22.221.126:45885)
0926 07:38:21.763542 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:21.763547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:21.764948 (+ 1401us) server_negotiation.cc:407] Connection header received
0926 07:38:21.766788 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:21.766794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:21.766891 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:21.767013 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:21.768169 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.768889 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.770672 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.771223 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.771386 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:21.772936 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:21.772943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:21.772947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:21.772968 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:21.773005 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:21.773020 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:21.773110 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:21.773329 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 07:38:21.773461 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:38:21.774794 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.221.67:0
--local_ip_for_outbound_sockets=127.22.221.67
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45885
--builtin_ntp_servers=127.22.221.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:38:21.775098 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:21.763889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45885 (local address 127.22.221.66:37451)
0926 07:38:21.764782 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:21.764826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:21.766507 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:21.767306 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:21.767315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:21.767416 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:21.767994 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:21.768011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.769073 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.769077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:21.770500 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:21.770516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:21.771508 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:21.771518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:21.771824 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:21.772625 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:21.772627 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:21.772669 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:21.772754 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:21.773121 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:21.773128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:21.773134 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:38:21.773285 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:38:21.773708 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260926 07:38:21.776794 11110 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45885
I20260926 07:38:21.777169 11110 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:21.778097 11110 heartbeater.cc:511] Master 127.22.221.126:45885 requested a full tablet report, sending...
I20260926 07:38:21.779891 10793 ts_manager.cc:194] Registered new tserver with Master: ecbda375ec044c4ebc908d3b0a0c23f8 (127.22.221.66:41673)
I20260926 07:38:21.781204 10793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.66:37451
W20260926 07:38:22.103737 11113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:22.104072 11113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:22.104240 11113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:22.111459 11113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:22.111627 11113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:22.111805 11113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.221.67
I20260926 07:38:22.120877 11113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.221.67
--webserver_port=0
--tserver_master_addrs=127.22.221.126:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.221.67
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:22.122184 11113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:22.123982 11113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:22.136719 11118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:22.136801 11119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:22.136826 11121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:22.137733 11113 server_base.cc:1034] running on GCE node
I20260926 07:38:22.138417 11113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:22.139438 11113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:22.140645 11113 hybrid_clock.cc:648] HybridClock initialized: now 1790408302140600 us; error 35 us; skew 500 ppm
I20260926 07:38:22.141000 11113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:38:22.143194 11113 webserver.cc:460] Webserver started at http://127.22.221.67:32939/ using document root <none> and password file <none>
I20260926 07:38:22.143922 11113 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:22.144069 11113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:22.144415 11113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:22.147271 11113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data/instance:
uuid: "4be50989cd4849a78b0f19ddcf13a527"
format_stamp: "Formatted at 2026-09-26 07:38:22 on dist-test-slave-qlq8"
server_key: "0e984d46e7bc9722ad978d7331e047c8"
server_key_iv: "23a0da9eb31b7bb2b7fd596c305eef2c"
server_key_version: "encryptionkey@0"
I20260926 07:38:22.148072 11113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance:
uuid: "4be50989cd4849a78b0f19ddcf13a527"
format_stamp: "Formatted at 2026-09-26 07:38:22 on dist-test-slave-qlq8"
server_key: "0e984d46e7bc9722ad978d7331e047c8"
server_key_iv: "23a0da9eb31b7bb2b7fd596c305eef2c"
server_key_version: "encryptionkey@0"
I20260926 07:38:22.153225 11113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:38:22.156569 11127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:22.157791 11113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:22.157977 11113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal
uuid: "4be50989cd4849a78b0f19ddcf13a527"
format_stamp: "Formatted at 2026-09-26 07:38:22 on dist-test-slave-qlq8"
server_key: "0e984d46e7bc9722ad978d7331e047c8"
server_key_iv: "23a0da9eb31b7bb2b7fd596c305eef2c"
server_key_version: "encryptionkey@0"
I20260926 07:38:22.158191 11113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:22.189117 11113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:22.190029 11113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:22.190348 11113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:22.191546 11113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:38:22.193100 11113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:38:22.193226 11113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:22.193367 11113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:38:22.193475 11113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:22.226332 11113 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.67:35719
I20260926 07:38:22.226423 11239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.67:35719 every 8 connection(s)
I20260926 07:38:22.227700 11113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/data/info.pb
I20260926 07:38:22.232057 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11113
I20260926 07:38:22.232287 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal/instance
I20260926 07:38:22.232894 23413 external_mini_cluster.cc:1428] Setting key 24b2676ccd96bd0887bda7591bca6de2
I20260926 07:38:22.241317 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.230482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:52933 (local address 127.22.221.126:45885)
0926 07:38:22.230778 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.230784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.232321 (+ 1537us) server_negotiation.cc:407] Connection header received
0926 07:38:22.234733 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.234737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.234810 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.234935 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:22.236211 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.237090 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.238724 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.239150 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.239310 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.240531 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:22.240535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:22.240538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:22.240556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:22.240589 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:22.240600 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:22.240671 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.240997 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.241078 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 07:38:22.242300 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.231013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:45885 (local address 127.22.221.67:52933)
0926 07:38:22.232145 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:22.232205 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:22.234418 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:22.235142 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:22.235150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:22.235289 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:38:22.236065 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.236084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.237244 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.237248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.238602 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.238614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.239398 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.239404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.239598 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.240236 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:38:22.240238 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:38:22.240276 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:38:22.240351 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:38:22.240713 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:38:22.240723 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:38:22.240729 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:38:22.240934 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 07:38:22.241233 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":322,"threads_started":1}
I20260926 07:38:22.243770 11240 heartbeater.cc:347] Connected to a master server at 127.22.221.126:45885
I20260926 07:38:22.244093 11240 heartbeater.cc:464] Registering TS with master...
I20260926 07:38:22.244810 11240 heartbeater.cc:511] Master 127.22.221.126:45885 requested a full tablet report, sending...
I20260926 07:38:22.246309 10793 ts_manager.cc:194] Registered new tserver with Master: 4be50989cd4849a78b0f19ddcf13a527 (127.22.221.67:35719)
I20260926 07:38:22.247519 10793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.221.67:52933
I20260926 07:38:22.248397 23413 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:38:22.258550 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.253101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.22.221.126:45885)
0926 07:38:22.253247 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.253251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.253731 (+ 480us) server_negotiation.cc:407] Connection header received
0926 07:38:22.253851 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.253855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.253921 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.254000 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:22.254439 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.255130 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.256519 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.256918 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.257008 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.258014 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:22.258019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:22.258022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:22.258038 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:22.258068 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:22.258080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:22.258156 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.258276 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.258347 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:38:22.262944 10792 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:38:22.269677 10792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:38:22.295367 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.281710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:44359 (local address 127.0.0.1:56784)
0926 07:38:22.286094 (+ 4384us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:22.286111 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:22.286222 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:22.288308 (+ 2086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:22.288320 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:22.288361 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:22.288916 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.288932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.290825 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.290830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.292897 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.292910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.294235 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.294241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.294574 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.294636 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:38:22.294772 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 07:38:22.294969 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4246,"mutex_wait_us":108,"thread_start_us":329,"threads_started":1}
I20260926 07:38:22.296108 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.282206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.22.221.65:44359)
0926 07:38:22.287441 (+ 5235us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.287454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.287493 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:38:22.287632 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.287643 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.287976 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.288240 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:22.289469 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.290671 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.293093 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.294024 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.294189 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.294448 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.295615 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.295801 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5054,"thread_start_us":4878,"threads_started":1}
I20260926 07:38:22.297646 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.281503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41673 (local address 127.0.0.1:47478)
0926 07:38:22.282392 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:22.282437 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:22.282600 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:22.289512 (+ 6912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:22.289516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:22.289543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:22.289999 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.290012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.292078 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.292087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.294520 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.294534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.296807 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.296813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.296986 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.297091 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:38:22.297174 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:38:22.297327 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":126,"thread_start_us":4303,"threads_started":1}
I20260926 07:38:22.297760 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.281676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.22.221.66:41673)
0926 07:38:22.288924 (+ 7248us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.288933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.288962 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:38:22.289063 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.289072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.289303 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.290186 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:22.290398 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.291947 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.294826 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.296308 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.296485 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.296719 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.297245 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.297406 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7080,"thread_start_us":6955,"threads_started":1}
I20260926 07:38:22.297843 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.286642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.22.221.67:35719)
0926 07:38:22.289278 (+ 2636us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.289286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.289315 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:38:22.289408 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.289417 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.289619 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.289821 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:22.290678 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.291810 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.295017 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.296161 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.296322 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.296622 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.297297 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.297468 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2472,"thread_start_us":2320,"threads_started":1}
I20260926 07:38:22.298597 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.286523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.67:35719 (local address 127.0.0.1:40838)
0926 07:38:22.287220 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:22.287274 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:22.287457 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:22.290080 (+ 2623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:22.290084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:22.290111 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:22.290466 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.290479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.291949 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.291955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.294810 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.294820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.296771 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.296779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.296983 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.297086 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 07:38:22.297174 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:38:22.298346 (+ 1172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1195,"thread_start_us":286,"threads_started":1}
I20260926 07:38:22.300585 10914 tablet_service.cc:1467] Processing CreateTablet for tablet 874624d6a6774cef9be942bee7b9c1f6 (DEFAULT_TABLE table=test-table [id=5ee940b5b20a46eb958039287b2f1aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:22.301375 11045 tablet_service.cc:1467] Processing CreateTablet for tablet 874624d6a6774cef9be942bee7b9c1f6 (DEFAULT_TABLE table=test-table [id=5ee940b5b20a46eb958039287b2f1aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:22.302115 10914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874624d6a6774cef9be942bee7b9c1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:22.302692 11045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874624d6a6774cef9be942bee7b9c1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:22.306566 11175 tablet_service.cc:1467] Processing CreateTablet for tablet 874624d6a6774cef9be942bee7b9c1f6 (DEFAULT_TABLE table=test-table [id=5ee940b5b20a46eb958039287b2f1aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:38:22.308089 11175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874624d6a6774cef9be942bee7b9c1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:22.317307 11258 tablet_bootstrap.cc:492] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Bootstrap starting.
I20260926 07:38:22.319360 11259 tablet_bootstrap.cc:492] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Bootstrap starting.
I20260926 07:38:22.321137 11258 tablet_bootstrap.cc:654] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:22.322772 11258 log.cc:824] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:22.323005 11259 tablet_bootstrap.cc:654] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:22.324285 11259 log.cc:824] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:22.325358 10979 heartbeater.cc:503] Master 127.22.221.126:45885 was elected leader, sending a full tablet report...
I20260926 07:38:22.326411 11258 tablet_bootstrap.cc:492] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: No bootstrap required, opened a new log
I20260926 07:38:22.326753 11258 ts_tablet_manager.cc:1397] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260926 07:38:22.331141 11260 tablet_bootstrap.cc:492] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Bootstrap starting.
I20260926 07:38:22.333484 11259 tablet_bootstrap.cc:492] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: No bootstrap required, opened a new log
I20260926 07:38:22.333818 11259 ts_tablet_manager.cc:1397] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260926 07:38:22.334473 11260 tablet_bootstrap.cc:654] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:22.335522 11260 log.cc:824] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:22.338214 11260 tablet_bootstrap.cc:492] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: No bootstrap required, opened a new log
I20260926 07:38:22.338485 11260 ts_tablet_manager.cc:1397] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260926 07:38:22.338208 11258 raft_consensus.cc:348] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.338654 11258 raft_consensus.cc:374] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:22.338766 11258 raft_consensus.cc:729] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbda375ec044c4ebc908d3b0a0c23f8, State: Initialized, Role: FOLLOWER
I20260926 07:38:22.339430 11258 consensus_queue.cc:260] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.340572 11110 heartbeater.cc:503] Master 127.22.221.126:45885 was elected leader, sending a full tablet report...
I20260926 07:38:22.341274 11258 ts_tablet_manager.cc:1428] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260926 07:38:22.344228 11259 raft_consensus.cc:348] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.344709 11259 raft_consensus.cc:374] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:22.344825 11259 raft_consensus.cc:729] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0ac665e57b459da291084a1fcbdd26, State: Initialized, Role: FOLLOWER
I20260926 07:38:22.345489 11259 consensus_queue.cc:260] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.347379 11259 ts_tablet_manager.cc:1428] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 07:38:22.348006 11260 raft_consensus.cc:348] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.348453 11260 raft_consensus.cc:374] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:22.348562 11260 raft_consensus.cc:729] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be50989cd4849a78b0f19ddcf13a527, State: Initialized, Role: FOLLOWER
I20260926 07:38:22.349118 11260 consensus_queue.cc:260] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.350226 11240 heartbeater.cc:503] Master 127.22.221.126:45885 was elected leader, sending a full tablet report...
I20260926 07:38:22.350816 11260 ts_tablet_manager.cc:1428] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260926 07:38:22.479504 11241 tablet.cc:2369] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:38:22.513077 11111 tablet.cc:2369] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:38:22.553483 10980 tablet.cc:2369] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:38:22.611509 11266 raft_consensus.cc:482] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:38:22.611801 11266 raft_consensus.cc:504] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.613559 11266 leader_election.cc:288] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecbda375ec044c4ebc908d3b0a0c23f8 (127.22.221.66:41673), bb0ac665e57b459da291084a1fcbdd26 (127.22.221.65:44359)
I20260926 07:38:22.620077 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.614031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:38283 (local address 127.22.221.66:41673)
0926 07:38:22.614247 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.614256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.614292 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:38:22.614389 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.614397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.614501 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.614610 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:22.615115 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.616016 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.618188 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.619087 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.619272 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.619344 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.619661 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.619834 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:38:22.620071 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.613829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.66:41673 (local address 127.22.221.67:38283)
0926 07:38:22.614066 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:22.614087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:22.614221 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:22.614679 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:22.614684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:22.614712 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:22.614947 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.614960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.616164 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.616168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.618030 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.618044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.619322 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.619327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.619481 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.619506 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:38:22.619629 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:38:22.619769 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 07:38:22.620891 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.613852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.65:44359 (local address 127.22.221.67:50177)
0926 07:38:22.614837 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0926 07:38:22.614854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:38:22.614980 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:38:22.615372 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:38:22.615376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:38:22.615396 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:38:22.615715 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.615762 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.616875 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.616881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.618599 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:38:22.618609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.620032 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.620038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:38:22.620244 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.620289 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:38:22.620460 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 07:38:22.620594 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":190,"thread_start_us":406,"threads_started":1}
I20260926 07:38:22.620910 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.614072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.221.67:50177 (local address 127.22.221.65:44359)
0926 07:38:22.614308 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.614314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.614970 (+ 656us) server_negotiation.cc:407] Connection header received
0926 07:38:22.615128 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.615133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.615250 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.615351 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:38:22.615917 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.616678 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.618784 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.619858 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.620024 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.620087 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.620520 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.620634 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:38:22.621304 11065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874624d6a6774cef9be942bee7b9c1f6" candidate_uuid: "4be50989cd4849a78b0f19ddcf13a527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" is_pre_election: true
I20260926 07:38:22.621886 11065 raft_consensus.cc:2454] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4be50989cd4849a78b0f19ddcf13a527 in term 0.
I20260926 07:38:22.621896 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874624d6a6774cef9be942bee7b9c1f6" candidate_uuid: "4be50989cd4849a78b0f19ddcf13a527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0ac665e57b459da291084a1fcbdd26" is_pre_election: true
I20260926 07:38:22.622442 10934 raft_consensus.cc:2454] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4be50989cd4849a78b0f19ddcf13a527 in term 0.
I20260926 07:38:22.622519 11131 leader_election.cc:302] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4be50989cd4849a78b0f19ddcf13a527, ecbda375ec044c4ebc908d3b0a0c23f8; no voters:
I20260926 07:38:22.622994 11266 raft_consensus.cc:2781] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:38:22.623138 11266 raft_consensus.cc:482] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:38:22.623265 11266 raft_consensus.cc:3037] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:22.625378 11266 raft_consensus.cc:504] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.625962 11266 leader_election.cc:288] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [CANDIDATE]: Term 1 election: Requested vote from peers ecbda375ec044c4ebc908d3b0a0c23f8 (127.22.221.66:41673), bb0ac665e57b459da291084a1fcbdd26 (127.22.221.65:44359)
I20260926 07:38:22.626340 11065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874624d6a6774cef9be942bee7b9c1f6" candidate_uuid: "4be50989cd4849a78b0f19ddcf13a527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8"
I20260926 07:38:22.626370 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874624d6a6774cef9be942bee7b9c1f6" candidate_uuid: "4be50989cd4849a78b0f19ddcf13a527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0ac665e57b459da291084a1fcbdd26"
I20260926 07:38:22.626555 11065 raft_consensus.cc:3037] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:22.626554 10934 raft_consensus.cc:3037] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:22.629760 11065 raft_consensus.cc:2454] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4be50989cd4849a78b0f19ddcf13a527 in term 1.
I20260926 07:38:22.629758 10934 raft_consensus.cc:2454] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4be50989cd4849a78b0f19ddcf13a527 in term 1.
I20260926 07:38:22.630168 11130 leader_election.cc:302] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4be50989cd4849a78b0f19ddcf13a527, bb0ac665e57b459da291084a1fcbdd26; no voters:
I20260926 07:38:22.630515 11266 raft_consensus.cc:2781] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:22.630961 11266 raft_consensus.cc:686] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 1 LEADER]: Becoming Leader. State: Replica: 4be50989cd4849a78b0f19ddcf13a527, State: Running, Role: LEADER
I20260926 07:38:22.631494 11266 consensus_queue.cc:237] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } }
I20260926 07:38:22.635833 10792 catalog_manager.cc:5495] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4be50989cd4849a78b0f19ddcf13a527 (127.22.221.67). New cstate: current_term: 1 leader_uuid: "4be50989cd4849a78b0f19ddcf13a527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4be50989cd4849a78b0f19ddcf13a527" member_type: VOTER last_known_addr { host: "127.22.221.67" port: 35719 } health_report { overall_health: HEALTHY } } }
I20260926 07:38:22.661252 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.655664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.22.221.67:35719)
0926 07:38:22.655872 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.655879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.655977 (+ 98us) server_negotiation.cc:407] Connection header received
0926 07:38:22.656226 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.656230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.656311 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.656473 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:38:22.656857 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.657604 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.659018 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.659502 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.659632 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.660152 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:38:22.660674 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:38:22.660852 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.660905 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.660995 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:38:22.669199 10934 raft_consensus.cc:1261] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 1 FOLLOWER]: Refusing update from remote peer 4be50989cd4849a78b0f19ddcf13a527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:22.669260 11065 raft_consensus.cc:1261] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 1 FOLLOWER]: Refusing update from remote peer 4be50989cd4849a78b0f19ddcf13a527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:38:22.670228 11268 consensus_queue.cc:1035] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecbda375ec044c4ebc908d3b0a0c23f8" member_type: VOTER last_known_addr { host: "127.22.221.66" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:22.670604 11266 consensus_queue.cc:1035] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0ac665e57b459da291084a1fcbdd26" member_type: VOTER last_known_addr { host: "127.22.221.65" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:38:22.682439 11271 mvcc.cc:205] Tried to move back new op lower bound from 7333512407723679744 to 7333512407582425088. Current Snapshot: MvccSnapshot[applied={T|T < 7333512407723679744}]
I20260926 07:38:22.696662 10792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:22.696915 10792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:38:22.700867 10792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 325137589cd14cf79fe2b062aef3f65f: Sending DeleteTablet for 3 replicas of tablet 874624d6a6774cef9be942bee7b9c1f6
I20260926 07:38:22.701972 11045 tablet_service.cc:1514] Processing DeleteTablet for tablet 874624d6a6774cef9be942bee7b9c1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:22 UTC) from {username='slave'} at 127.0.0.1:47478
I20260926 07:38:22.702083 10914 tablet_service.cc:1514] Processing DeleteTablet for tablet 874624d6a6774cef9be942bee7b9c1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:22 UTC) from {username='slave'} at 127.0.0.1:56784
I20260926 07:38:22.703111 11175 tablet_service.cc:1514] Processing DeleteTablet for tablet 874624d6a6774cef9be942bee7b9c1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:38:22 UTC) from {username='slave'} at 127.0.0.1:40838
I20260926 07:38:22.703608 11282 tablet_replica.cc:330] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: stopping tablet replica
I20260926 07:38:22.703605 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10852
I20260926 07:38:22.704129 11282 raft_consensus.cc:2229] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:22.704728 11282 raft_consensus.cc:2258] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:22.705540 11281 tablet_replica.cc:330] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: stopping tablet replica
I20260926 07:38:22.706092 11281 raft_consensus.cc:2229] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:38:22.706723 11281 raft_consensus.cc:2258] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:22.708256 11282 ts_tablet_manager.cc:1905] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:22.712538 11281 ts_tablet_manager.cc:1905] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:22.716734 11282 ts_tablet_manager.cc:1918] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:22.717206 11282 log.cc:1196] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-0/wal/wals/874624d6a6774cef9be942bee7b9c1f6
I20260926 07:38:22.717655 11283 tablet_replica.cc:330] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: stopping tablet replica
I20260926 07:38:22.717984 11282 ts_tablet_manager.cc:1939] T 874624d6a6774cef9be942bee7b9c1f6 P bb0ac665e57b459da291084a1fcbdd26: Deleting consensus metadata
I20260926 07:38:22.718087 11283 raft_consensus.cc:2229] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:38:22.718690 11283 raft_consensus.cc:2258] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:38:22.720165 10779 catalog_manager.cc:4841] TS bb0ac665e57b459da291084a1fcbdd26 (127.22.221.65:44359): tablet 874624d6a6774cef9be942bee7b9c1f6 (table test-table [id=5ee940b5b20a46eb958039287b2f1aa7]) successfully deleted
I20260926 07:38:22.721441 11283 ts_tablet_manager.cc:1905] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:38:22.723658 11281 ts_tablet_manager.cc:1918] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:22.723886 11281 log.cc:1196] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-1/wal/wals/874624d6a6774cef9be942bee7b9c1f6
I20260926 07:38:22.724479 11281 ts_tablet_manager.cc:1939] T 874624d6a6774cef9be942bee7b9c1f6 P ecbda375ec044c4ebc908d3b0a0c23f8: Deleting consensus metadata
I20260926 07:38:22.726271 10780 catalog_manager.cc:4841] TS ecbda375ec044c4ebc908d3b0a0c23f8 (127.22.221.66:41673): tablet 874624d6a6774cef9be942bee7b9c1f6 (table test-table [id=5ee940b5b20a46eb958039287b2f1aa7]) successfully deleted
I20260926 07:38:22.728899 11283 ts_tablet_manager.cc:1918] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:38:22.729095 11283 log.cc:1196] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Deleting WAL directory at /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407989013554-23413-0/minicluster-data/ts-2/wal/wals/874624d6a6774cef9be942bee7b9c1f6
I20260926 07:38:22.729527 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.704753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.22.221.65:44359)
0926 07:38:22.704919 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.704925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.721560 (+ 16635us) server_negotiation.cc:407] Connection header received
0926 07:38:22.722015 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.722022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.722118 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.722185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:22.722892 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.724061 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.726146 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.726738 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.726812 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.728156 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:22.728173 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:22.728186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:22.728750 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:22.728795 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:22.728814 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:22.728991 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.729047 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.729157 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:38:22.729688 11283 ts_tablet_manager.cc:1939] T 874624d6a6774cef9be942bee7b9c1f6 P 4be50989cd4849a78b0f19ddcf13a527: Deleting consensus metadata
I20260926 07:38:22.730264 10974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:22.731400 10780 catalog_manager.cc:4841] TS 4be50989cd4849a78b0f19ddcf13a527 (127.22.221.67:35719): tablet 874624d6a6774cef9be942bee7b9c1f6 (table test-table [id=5ee940b5b20a46eb958039287b2f1aa7]) successfully deleted
I20260926 07:38:22.873256 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10852
I20260926 07:38:22.891588 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10983
I20260926 07:38:22.898785 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:22.892442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.22.221.66:41673)
0926 07:38:22.892621 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:22.892629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:22.892652 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:38:22.892878 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:22.892884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:22.892991 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:22.893103 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:22.893533 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.894261 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.895621 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:22.896049 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:22.896180 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:22.897307 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:22.897324 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:22.897338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:22.897970 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:22.898019 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:22.898038 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:22.898168 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:22.898309 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 07:38:22.898420 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:38:22.899506 11105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:22.990866 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10983
I20260926 07:38:23.014247 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11113
I20260926 07:38:23.022024 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:23.015170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.22.221.67:35719)
0926 07:38:23.015434 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:23.015440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:23.015465 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:38:23.015566 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:23.015571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:23.015663 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:23.015798 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:23.016193 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:23.017011 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:23.018536 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:23.019004 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:23.019159 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:23.020413 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:23.020425 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:23.020440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:23.021112 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:23.021160 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:23.021180 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:23.021331 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:23.021463 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:38:23.021567 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 07:38:23.022778 11235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:23.144850 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11113
I20260926 07:38:23.164439 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 10762
I20260926 07:38:23.165664 10823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:23.277736 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 10762
2026-09-26T07:38:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3069 ms)
[----------] 6 tests from EncryptionPolicyTest (22035 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:23 dist-test-slave-qlq8 krb5kdc[11291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:23 dist-test-slave-qlq8 krb5kdc[11291](info): set up 2 sockets
Sep 26 07:38:23 dist-test-slave-qlq8 krb5kdc[11291](info): commencing operation
krb5kdc: starting...
W20260926 07:38:24.399842 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.009s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:24 dist-test-slave-qlq8 krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:24.665357 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:25.001291 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:25.001662 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:25.001947 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:25.009402 11304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:25.009584 11304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:25.009711 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:25.009821 11304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:25.009902 11304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:25.019882 11304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:25.021271 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:25.022624 11304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:25.034497 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:25.034551 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:25.034574 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:25.035703 11304 server_base.cc:1034] running on GCE node
I20260926 07:38:25.036569 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:25.038024 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:25.039336 11304 hybrid_clock.cc:648] HybridClock initialized: now 1790408305039268 us; error 48 us; skew 500 ppm
I20260926 07:38:25.039705 11304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:25 dist-test-slave-qlq8 krb5kdc[11291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:25.042896 11304 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:25.044472 11304 webserver.cc:460] Webserver started at http://127.0.0.1:38023/ using document root <none> and password file <none>
I20260926 07:38:25.045156 11304 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:25.045305 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:25.045609 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:25.048722 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2"
format_stamp: "Formatted at 2026-09-26 07:38:25 on dist-test-slave-qlq8"
server_key: "96d679a68c12957aa9fa0ca8d8787faf"
server_key_iv: "1477ce7dc1de8a90ac38a2b9d6d192b9"
server_key_version: "encryptionkey@0"
I20260926 07:38:25.049521 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2"
format_stamp: "Formatted at 2026-09-26 07:38:25 on dist-test-slave-qlq8"
server_key: "96d679a68c12957aa9fa0ca8d8787faf"
server_key_iv: "1477ce7dc1de8a90ac38a2b9d6d192b9"
server_key_version: "encryptionkey@0"
I20260926 07:38:25.054814 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:25.058486 11319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:25.060034 11304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:25.060237 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2"
format_stamp: "Formatted at 2026-09-26 07:38:25 on dist-test-slave-qlq8"
server_key: "96d679a68c12957aa9fa0ca8d8787faf"
server_key_iv: "1477ce7dc1de8a90ac38a2b9d6d192b9"
server_key_version: "encryptionkey@0"
I20260926 07:38:25.060460 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:25.075918 11304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:25.080274 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:25.080574 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:25.099182 11304 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45035
I20260926 07:38:25.099275 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45035 every 8 connection(s)
I20260926 07:38:25.100582 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:25.104338 11371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:25.109905 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11304
I20260926 07:38:25.110113 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:25.110951 23413 external_mini_cluster.cc:1428] Setting key bcfc538ca638bf5083d02682f2525585
I20260926 07:38:25.116325 11371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2: Bootstrap starting.
I20260926 07:38:25.120393 11371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:25.121840 11371 log.cc:824] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:25.125456 11371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2: No bootstrap required, opened a new log
Sep 26 07:38:25 dist-test-slave-qlq8 krb5kdc[11291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:38:25.135305 11371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260926 07:38:25.135769 11371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:25.135890 11371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8218e69f814d5bb9ea9ae2ff308dc2, State: Initialized, Role: FOLLOWER
I20260926 07:38:25.136650 11371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260926 07:38:25.136950 11371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:25.137105 11371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:25.137286 11371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:25.139125 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:25.113462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57527 (local address 127.0.0.1:45035)
0926 07:38:25.114494 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:25.114509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:25.114975 (+ 466us) server_negotiation.cc:407] Connection header received
0926 07:38:25.117472 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:25.117500 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:25.118052 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:25.118587 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:25.120145 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:25.122253 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:25.124092 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:25.125168 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:25.125375 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:25.129738 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:25.129754 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:25.129780 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:25.129832 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:25.134080 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:25.134959 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:25.134968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:25.134976 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:25.135179 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:25.135695 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:25.135699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:25.135702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:25.136435 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:25.136773 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:25.137818 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0926 07:38:25.138077 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":440,"threads_started":1}
I20260926 07:38:25.139879 11371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260926 07:38:25.140394 11371 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f8218e69f814d5bb9ea9ae2ff308dc2; no voters:
I20260926 07:38:25.141064 11371 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:25.141177 11376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:25.142338 11376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 2f8218e69f814d5bb9ea9ae2ff308dc2, State: Running, Role: LEADER
I20260926 07:38:25.142966 11376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260926 07:38:25.143499 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:25.143524 11371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:25.146153 11377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } } }
I20260926 07:38:25.146590 11377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:25.151207 11378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8218e69f814d5bb9ea9ae2ff308dc2. Latest consensus state: current_term: 1 leader_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8218e69f814d5bb9ea9ae2ff308dc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } } }
I20260926 07:38:25.152244 11378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:25.158221 11382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:25.168133 11382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:25.186609 11382 catalog_manager.cc:1347] Generated new cluster ID: 43d5febfe9ae482ba54b151ed5aa5701
I20260926 07:38:25.186748 11382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:25.191020 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:25.159874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33585 (local address 127.0.0.1:45035)
0926 07:38:25.160106 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:25.160112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:25.172009 (+ 11897us) server_negotiation.cc:407] Connection header received
0926 07:38:25.172230 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:25.172235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:25.172325 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:25.172442 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:25.174635 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:25.175727 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:25.178086 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:25.179001 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:25.179117 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:25.183955 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:25.183961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:25.183966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:25.184006 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:25.187830 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:25.188801 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:25.188807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:25.188811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:25.188935 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:25.189456 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:25.189461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:25.189464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:25.189814 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:25.190036 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:25.190477 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 07:38:25.190640 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 07:38:25.205421 11382 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:25.206751 11382 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:25.218698 11382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f8218e69f814d5bb9ea9ae2ff308dc2: Generated new TSK 0
I20260926 07:38:25.219563 11382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:25.227836 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11304
I20260926 07:38:25.229565 11366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:25.321233 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11304
W20260926 07:38:25.338330 11289 connection.cc:535] client connection to 127.0.0.1:45035 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45035): Connection reset by peer (error 104)
2026-09-26T07:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2038 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:25 dist-test-slave-qlq8 krb5kdc[11403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:25 dist-test-slave-qlq8 krb5kdc[11403](info): set up 2 sockets
Sep 26 07:38:25 dist-test-slave-qlq8 krb5kdc[11403](info): commencing operation
krb5kdc: starting...
W20260926 07:38:26.423746 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:26 dist-test-slave-qlq8 krb5kdc[11403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:26.683131 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:27.010969 11416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:27.011412 11416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:27.011564 11416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:27.018715 11416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:27.018826 11416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:27.018924 11416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:27.019021 11416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:27.019081 11416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:27.028921 11416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:27.030191 11416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:27.031522 11416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:27.043776 11422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:27.043829 11421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:27.043800 11424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:27.044734 11416 server_base.cc:1034] running on GCE node
I20260926 07:38:27.045595 11416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:27.047115 11416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:27.048359 11416 hybrid_clock.cc:648] HybridClock initialized: now 1790408307048298 us; error 56 us; skew 500 ppm
I20260926 07:38:27.048722 11416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:27 dist-test-slave-qlq8 krb5kdc[11403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:27.056279 11416 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:27.058086 11416 webserver.cc:460] Webserver started at http://127.0.0.1:42893/ using document root <none> and password file <none>
I20260926 07:38:27.058847 11416 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:27.058997 11416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:27.059379 11416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:27.062422 11416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "658505de954c49869eadc7131292bf4a"
format_stamp: "Formatted at 2026-09-26 07:38:27 on dist-test-slave-qlq8"
server_key: "8e43e5dd16387099b318f8864a9a1ec5"
server_key_iv: "b53a6eaecae026e2bb4e0bfa1cbff038"
server_key_version: "encryptionkey@0"
I20260926 07:38:27.063329 11416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "658505de954c49869eadc7131292bf4a"
format_stamp: "Formatted at 2026-09-26 07:38:27 on dist-test-slave-qlq8"
server_key: "8e43e5dd16387099b318f8864a9a1ec5"
server_key_iv: "b53a6eaecae026e2bb4e0bfa1cbff038"
server_key_version: "encryptionkey@0"
I20260926 07:38:27.068655 11416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:38:27.072175 11431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:27.073663 11416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:38:27.073896 11416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "658505de954c49869eadc7131292bf4a"
format_stamp: "Formatted at 2026-09-26 07:38:27 on dist-test-slave-qlq8"
server_key: "8e43e5dd16387099b318f8864a9a1ec5"
server_key_iv: "b53a6eaecae026e2bb4e0bfa1cbff038"
server_key_version: "encryptionkey@0"
I20260926 07:38:27.074107 11416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:27.089950 11416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:27.094089 11416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:27.094372 11416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:27.113153 11416 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44341
I20260926 07:38:27.113206 11482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44341 every 8 connection(s)
I20260926 07:38:27.114607 11416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:27.116878 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11416
I20260926 07:38:27.117081 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:27.117580 23413 external_mini_cluster.cc:1428] Setting key a469cff73c125ab39932d2ac60b034ef
I20260926 07:38:27.119098 11483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:27.133000 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:27.119536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59537 (local address 127.0.0.1:44341)
0926 07:38:27.120346 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:27.120362 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:27.120411 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:38:27.122434 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:27.122461 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:27.123030 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:27.123510 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:27.124048 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:27.126120 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:27.127582 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:27.128621 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:27.128817 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:27.130034 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:27.130045 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:27.130064 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:27.130841 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:27.130941 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:27.130978 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:27.131146 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:27.131500 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 07:38:27.131686 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":331,"threads_started":1}
I20260926 07:38:27.133989 11483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a: Bootstrap starting.
I20260926 07:38:27.137063 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:27.137403 11483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:27.138633 11483 log.cc:824] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:27.141990 11483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a: No bootstrap required, opened a new log
I20260926 07:38:27.149225 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:27.142360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50279 (local address 127.0.0.1:44341)
0926 07:38:27.142564 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:27.142570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:27.142997 (+ 427us) server_negotiation.cc:407] Connection header received
0926 07:38:27.143148 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:27.143154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:27.143262 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:27.143378 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:27.143836 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:27.145065 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:27.146392 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:27.147113 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:27.147284 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:27.148384 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:27.148390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:27.148396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:27.148416 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:27.148460 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:27.148490 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:27.148605 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:27.148847 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:38:27.148898 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:38:27.151805 11483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658505de954c49869eadc7131292bf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260926 07:38:27.152168 11483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:27.152274 11483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658505de954c49869eadc7131292bf4a, State: Initialized, Role: FOLLOWER
I20260926 07:38:27.152842 11483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658505de954c49869eadc7131292bf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260926 07:38:27.153074 11483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:27.153239 11483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:27.153405 11483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:27.155911 11483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658505de954c49869eadc7131292bf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260926 07:38:27.156419 11483 leader_election.cc:302] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 658505de954c49869eadc7131292bf4a; no voters:
I20260926 07:38:27.156977 11483 leader_election.cc:288] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:27.157159 11493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:27.157858 11493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [term 1 LEADER]: Becoming Leader. State: Replica: 658505de954c49869eadc7131292bf4a, State: Running, Role: LEADER
I20260926 07:38:27.158413 11493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658505de954c49869eadc7131292bf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260926 07:38:27.159024 11483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:27.162168 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "658505de954c49869eadc7131292bf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658505de954c49869eadc7131292bf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260926 07:38:27.162155 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658505de954c49869eadc7131292bf4a. Latest consensus state: current_term: 1 leader_uuid: "658505de954c49869eadc7131292bf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "658505de954c49869eadc7131292bf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260926 07:38:27.162571 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:27.162621 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:27.163448 11501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:27.168573 11501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:27.176895 11501 catalog_manager.cc:1347] Generated new cluster ID: 2a1ee4b330914fd4bcefd99b49a1f738
I20260926 07:38:27.177032 11501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:27.187883 11501 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:27.189112 11501 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:27.195057 11501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 658505de954c49869eadc7131292bf4a: Generated new TSK 0
I20260926 07:38:27.195906 11501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:27.201953 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11416
W20260926 07:38:27.203428 11478 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59537
W20260926 07:38:27.204193 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:27.204382 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11416
W20260926 07:38:27.221095 11401 connection.cc:535] client connection to 127.0.0.1:44341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44341): Connection reset by peer (error 104)
2026-09-26T07:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:27 dist-test-slave-qlq8 krb5kdc[11514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:27 dist-test-slave-qlq8 krb5kdc[11514](info): set up 2 sockets
Sep 26 07:38:27 dist-test-slave-qlq8 krb5kdc[11514](info): commencing operation
krb5kdc: starting...
W20260926 07:38:28.328326 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:28 dist-test-slave-qlq8 krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:28.603816 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:28.936539 11527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:28.936841 11527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:28.936934 11527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:28.944290 11527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:28.944398 11527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:28.944465 11527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:28.944525 11527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:28.944571 11527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:28.954372 11527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:28.955607 11527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:28.956893 11527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:28.968466 11532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:28.968432 11533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:28.968394 11535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:28.969514 11527 server_base.cc:1034] running on GCE node
I20260926 07:38:28.970363 11527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:28.971894 11527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:28.973134 11527 hybrid_clock.cc:648] HybridClock initialized: now 1790408308973089 us; error 38 us; skew 500 ppm
I20260926 07:38:28.973500 11527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:28 dist-test-slave-qlq8 krb5kdc[11514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:28.976643 11527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:28.978163 11527 webserver.cc:460] Webserver started at http://127.0.0.1:34213/ using document root <none> and password file <none>
I20260926 07:38:28.978834 11527 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:28.978981 11527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:28.979363 11527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:28.982268 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "f916c7dbd5914836a6b650d33ced802e"
format_stamp: "Formatted at 2026-09-26 07:38:28 on dist-test-slave-qlq8"
server_key: "a87fa614426d48cfc587faf5469be839"
server_key_iv: "77a53d00327bafc76292d3e4e1a7ca32"
server_key_version: "encryptionkey@0"
I20260926 07:38:28.983035 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "f916c7dbd5914836a6b650d33ced802e"
format_stamp: "Formatted at 2026-09-26 07:38:28 on dist-test-slave-qlq8"
server_key: "a87fa614426d48cfc587faf5469be839"
server_key_iv: "77a53d00327bafc76292d3e4e1a7ca32"
server_key_version: "encryptionkey@0"
I20260926 07:38:28.988058 11527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 07:38:28.991395 11542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:28.992635 11527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:38:28.992831 11527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "f916c7dbd5914836a6b650d33ced802e"
format_stamp: "Formatted at 2026-09-26 07:38:28 on dist-test-slave-qlq8"
server_key: "a87fa614426d48cfc587faf5469be839"
server_key_iv: "77a53d00327bafc76292d3e4e1a7ca32"
server_key_version: "encryptionkey@0"
I20260926 07:38:28.993021 11527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:28.998755 11527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:29.002689 11527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:29.002988 11527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:29.021616 11527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40571
I20260926 07:38:29.021670 11593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40571 every 8 connection(s)
I20260926 07:38:29.023030 11527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:29.026655 11594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:29.027482 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11527
I20260926 07:38:29.027649 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:29.028090 23413 external_mini_cluster.cc:1428] Setting key 82558c3e684762e5efadd0df6cb1c213
I20260926 07:38:29.036044 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:29.029933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46761 (local address 127.0.0.1:40571)
0926 07:38:29.030814 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:29.030828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:29.030871 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:38:29.032568 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:29.032588 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:29.032927 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:29.033249 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:29.034386 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:29.034392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:29.034405 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:29.034420 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:29.034502 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:29.034524 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:29.034742 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:29.034843 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:38:29.035065 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":389,"threads_started":1}
I20260926 07:38:29.039937 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:29.040055 11594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e: Bootstrap starting.
I20260926 07:38:29.043020 11594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:29.044171 11594 log.cc:824] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:29.047300 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:29.044425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47889 (local address 127.0.0.1:40571)
0926 07:38:29.044599 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:29.044605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:29.045042 (+ 437us) server_negotiation.cc:407] Connection header received
0926 07:38:29.045222 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:29.045226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:29.045273 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:29.045351 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:29.046532 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:29.046537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:29.046541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:29.046559 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:29.046607 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:29.046631 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:29.046760 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:29.046936 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:38:29.046998 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:38:29.047621 11594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e: No bootstrap required, opened a new log
I20260926 07:38:29.055488 11594 raft_consensus.cc:348] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916c7dbd5914836a6b650d33ced802e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260926 07:38:29.055846 11594 raft_consensus.cc:374] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:29.055991 11594 raft_consensus.cc:729] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f916c7dbd5914836a6b650d33ced802e, State: Initialized, Role: FOLLOWER
I20260926 07:38:29.056565 11594 consensus_queue.cc:260] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916c7dbd5914836a6b650d33ced802e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260926 07:38:29.056803 11594 raft_consensus.cc:388] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:29.056962 11594 raft_consensus.cc:482] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:29.057125 11594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:29.059417 11594 raft_consensus.cc:504] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916c7dbd5914836a6b650d33ced802e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260926 07:38:29.059969 11594 leader_election.cc:302] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f916c7dbd5914836a6b650d33ced802e; no voters:
I20260926 07:38:29.060734 11594 leader_election.cc:288] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:29.060847 11604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:29.061522 11604 raft_consensus.cc:686] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [term 1 LEADER]: Becoming Leader. State: Replica: f916c7dbd5914836a6b650d33ced802e, State: Running, Role: LEADER
I20260926 07:38:29.062186 11604 consensus_queue.cc:237] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916c7dbd5914836a6b650d33ced802e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260926 07:38:29.063148 11594 sys_catalog.cc:564] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:29.065442 11606 sys_catalog.cc:455] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f916c7dbd5914836a6b650d33ced802e. Latest consensus state: current_term: 1 leader_uuid: "f916c7dbd5914836a6b650d33ced802e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916c7dbd5914836a6b650d33ced802e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } } }
I20260926 07:38:29.065893 11606 sys_catalog.cc:458] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:29.066425 11605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f916c7dbd5914836a6b650d33ced802e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916c7dbd5914836a6b650d33ced802e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } } }
I20260926 07:38:29.066704 11605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:29.068181 11613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:29.074609 11613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:29.083727 11613 catalog_manager.cc:1347] Generated new cluster ID: 8cc2920d7594470b9bb2a8f0e59fc2a2
I20260926 07:38:29.083895 11613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:29.102500 11613 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:29.102949 11613 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:29.112039 11613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f916c7dbd5914836a6b650d33ced802e: Generated new TSK 0
I20260926 07:38:29.112931 11613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:29.123425 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11527
W20260926 07:38:29.124531 11589 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46761
W20260926 07:38:29.125288 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11527: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:29.125485 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11527
2026-09-26T07:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:29 dist-test-slave-qlq8 krb5kdc[11625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:29 dist-test-slave-qlq8 krb5kdc[11625](info): set up 2 sockets
Sep 26 07:38:29 dist-test-slave-qlq8 krb5kdc[11625](info): commencing operation
krb5kdc: starting...
W20260926 07:38:30.227707 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:30 dist-test-slave-qlq8 krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:30.487424 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:30.818949 11638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:30.819242 11638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:30.819402 11638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:30.826493 11638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:30.826596 11638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:30.826645 11638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:30.826697 11638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:30.826733 11638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:30.837638 11638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:30.838935 11638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:30.840287 11638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:30.852008 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:30.852037 11644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:30.852294 11643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:30.852926 11638 server_base.cc:1034] running on GCE node
I20260926 07:38:30.853720 11638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:30.855119 11638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:30.856355 11638 hybrid_clock.cc:648] HybridClock initialized: now 1790408310856331 us; error 80 us; skew 500 ppm
I20260926 07:38:30.856719 11638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:30 dist-test-slave-qlq8 krb5kdc[11625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:30.859936 11638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:30.861532 11638 webserver.cc:460] Webserver started at http://127.0.0.1:41061/ using document root <none> and password file <none>
I20260926 07:38:30.862196 11638 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:30.862342 11638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:30.862644 11638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:30.865633 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "378f6f35c8d546529ad4f0a94db9f617"
format_stamp: "Formatted at 2026-09-26 07:38:30 on dist-test-slave-qlq8"
server_key: "374ed38c898d2950ee0f17febca0f3be"
server_key_iv: "97725bafeba61f8d692af2fce8780bed"
server_key_version: "encryptionkey@0"
I20260926 07:38:30.866403 11638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "378f6f35c8d546529ad4f0a94db9f617"
format_stamp: "Formatted at 2026-09-26 07:38:30 on dist-test-slave-qlq8"
server_key: "374ed38c898d2950ee0f17febca0f3be"
server_key_iv: "97725bafeba61f8d692af2fce8780bed"
server_key_version: "encryptionkey@0"
I20260926 07:38:30.871567 11638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 07:38:30.874948 11653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:30.876395 11638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:30.876610 11638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "378f6f35c8d546529ad4f0a94db9f617"
format_stamp: "Formatted at 2026-09-26 07:38:30 on dist-test-slave-qlq8"
server_key: "374ed38c898d2950ee0f17febca0f3be"
server_key_iv: "97725bafeba61f8d692af2fce8780bed"
server_key_version: "encryptionkey@0"
I20260926 07:38:30.876827 11638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:30.893850 11638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:30.897941 11638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:30.898236 11638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:30.917343 11638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35175
I20260926 07:38:30.917366 11704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35175 every 8 connection(s)
I20260926 07:38:30.918733 11638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:30.922467 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11638
I20260926 07:38:30.922537 11705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:30.922672 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:30.923271 23413 external_mini_cluster.cc:1428] Setting key 1d64f9a6a3a7037ac4253dd4968ad994
I20260926 07:38:30.936833 11705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617: Bootstrap starting.
Sep 26 07:38:30 dist-test-slave-qlq8 krb5kdc[11625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:38:30.940197 11705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:30.941336 11705 log.cc:824] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:30.944169 11705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617: No bootstrap required, opened a new log
I20260926 07:38:30.946154 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:30.925023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41765 (local address 127.0.0.1:35175)
0926 07:38:30.925861 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:30.925872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:30.925915 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:38:30.927601 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:30.927623 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:30.928027 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:30.928358 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:30.930130 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:30.931498 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:30.933279 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:30.934006 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:30.934135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:30.938798 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:30.938806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:30.938819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:30.938858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:30.942128 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:30.942844 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:30.942850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:30.942855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:30.943014 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:30.943474 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:30.943479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:30.943489 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:30.943984 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:30.944290 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:30.944785 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 07:38:30.945203 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":418,"threads_started":1}
I20260926 07:38:30.950088 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:30.952973 11705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f6f35c8d546529ad4f0a94db9f617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260926 07:38:30.953374 11705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:30.953503 11705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378f6f35c8d546529ad4f0a94db9f617, State: Initialized, Role: FOLLOWER
I20260926 07:38:30.954071 11705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f6f35c8d546529ad4f0a94db9f617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260926 07:38:30.954361 11705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:30.954527 11705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:30.954749 11705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:30.957250 11705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f6f35c8d546529ad4f0a94db9f617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260926 07:38:30.957815 11705 leader_election.cc:302] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 378f6f35c8d546529ad4f0a94db9f617; no voters:
I20260926 07:38:30.958526 11705 leader_election.cc:288] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:30.958629 11715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:30.960320 11715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [term 1 LEADER]: Becoming Leader. State: Replica: 378f6f35c8d546529ad4f0a94db9f617, State: Running, Role: LEADER
I20260926 07:38:30.960992 11715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f6f35c8d546529ad4f0a94db9f617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260926 07:38:30.961180 11705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:30.970145 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:30.954763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50527 (local address 127.0.0.1:35175)
0926 07:38:30.954993 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:30.954999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:30.955575 (+ 576us) server_negotiation.cc:407] Connection header received
0926 07:38:30.955826 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:30.955830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:30.955900 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:30.956021 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:30.957485 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:30.958297 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:30.960110 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:30.960848 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:30.960985 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:30.963915 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:30.963925 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:30.963928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:30.963959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:30.966829 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:30.967678 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:30.967683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:30.967686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:30.967836 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:30.968217 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:30.968223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:30.968226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:30.968565 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:30.969114 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:30.969322 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:38:30.969634 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:38:30.972512 11716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "378f6f35c8d546529ad4f0a94db9f617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f6f35c8d546529ad4f0a94db9f617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260926 07:38:30.972867 11716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:30.973587 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 378f6f35c8d546529ad4f0a94db9f617. Latest consensus state: current_term: 1 leader_uuid: "378f6f35c8d546529ad4f0a94db9f617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "378f6f35c8d546529ad4f0a94db9f617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260926 07:38:30.973884 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:30.976806 11725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:30.981563 11725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:30.990434 11725 catalog_manager.cc:1347] Generated new cluster ID: 4f0f8fc74eb64f8ba5256a92779864cf
I20260926 07:38:30.990558 11725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:31.008466 11725 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:31.009747 11725 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:31.020355 11725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 378f6f35c8d546529ad4f0a94db9f617: Generated new TSK 0
I20260926 07:38:31.021210 11725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:31.029024 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11638
I20260926 07:38:31.030239 11700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:31.122104 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11638
2026-09-26T07:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:31 dist-test-slave-qlq8 krb5kdc[11737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:31 dist-test-slave-qlq8 krb5kdc[11737](info): set up 2 sockets
Sep 26 07:38:31 dist-test-slave-qlq8 krb5kdc[11737](info): commencing operation
krb5kdc: starting...
W20260926 07:38:32.235783 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:32 dist-test-slave-qlq8 krb5kdc[11737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:32.503882 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:32.840870 11750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:32.841239 11750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:32.841403 11750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:32.850692 11750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:32.850803 11750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:32.850927 11750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:32.851037 11750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:32.851122 11750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:32.861671 11750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:32.862969 11750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:32.864317 11750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:32.876528 11755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:32.876449 11758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:32.876436 11756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:32.877434 11750 server_base.cc:1034] running on GCE node
I20260926 07:38:32.878265 11750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:32.879873 11750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:32.881083 11750 hybrid_clock.cc:648] HybridClock initialized: now 1790408312881049 us; error 34 us; skew 500 ppm
I20260926 07:38:32.881458 11750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:32 dist-test-slave-qlq8 krb5kdc[11737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:32.888377 11750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:32.889966 11750 webserver.cc:460] Webserver started at http://127.0.0.1:35849/ using document root <none> and password file <none>
I20260926 07:38:32.890702 11750 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:32.890805 11750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:32.891145 11750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:32.894202 11750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "3294537c4ba54a208a675b571a992a9d"
format_stamp: "Formatted at 2026-09-26 07:38:32 on dist-test-slave-qlq8"
server_key: "0ff37f5dc6699b6ccc751660fcaa0d94"
server_key_iv: "b61f96358c2920c35bad56e61e68eb6a"
server_key_version: "encryptionkey@0"
I20260926 07:38:32.894976 11750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "3294537c4ba54a208a675b571a992a9d"
format_stamp: "Formatted at 2026-09-26 07:38:32 on dist-test-slave-qlq8"
server_key: "0ff37f5dc6699b6ccc751660fcaa0d94"
server_key_iv: "b61f96358c2920c35bad56e61e68eb6a"
server_key_version: "encryptionkey@0"
I20260926 07:38:32.900285 11750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 07:38:32.903894 11765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:32.905232 11750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:32.905422 11750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "3294537c4ba54a208a675b571a992a9d"
format_stamp: "Formatted at 2026-09-26 07:38:32 on dist-test-slave-qlq8"
server_key: "0ff37f5dc6699b6ccc751660fcaa0d94"
server_key_iv: "b61f96358c2920c35bad56e61e68eb6a"
server_key_version: "encryptionkey@0"
I20260926 07:38:32.905633 11750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:32.922991 11750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:32.927580 11750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:32.927896 11750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:32.948128 11750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39367
I20260926 07:38:32.948175 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39367 every 8 connection(s)
I20260926 07:38:32.949589 11750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:32.953398 11817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:32.959681 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11750
I20260926 07:38:32.959863 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:32.960417 23413 external_mini_cluster.cc:1428] Setting key 25d95577ec43b146e65f3c4ad68027be
I20260926 07:38:32.964627 11817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d: Bootstrap starting.
I20260926 07:38:32.968319 11817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:32.969574 11817 log.cc:824] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:32.973057 11817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d: No bootstrap required, opened a new log
I20260926 07:38:32.974586 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:32.962292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44985 (local address 127.0.0.1:39367)
0926 07:38:32.963088 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:32.963100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:32.963140 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:38:32.964985 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:32.965011 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:32.965406 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:32.965722 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:32.966316 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:32.967690 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:32.969728 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:32.970785 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:32.970982 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:32.972298 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:32.972305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:32.972320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:32.972853 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:32.972930 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:32.972960 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:32.973051 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:32.973347 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:38:32.973651 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":380,"threads_started":1}
I20260926 07:38:32.978561 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:32.981141 11817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294537c4ba54a208a675b571a992a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260926 07:38:32.981644 11817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:32.981791 11817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3294537c4ba54a208a675b571a992a9d, State: Initialized, Role: FOLLOWER
I20260926 07:38:32.982424 11817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294537c4ba54a208a675b571a992a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260926 07:38:32.982673 11817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:32.982841 11817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:32.983065 11817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:32.987609 11817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294537c4ba54a208a675b571a992a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260926 07:38:32.988340 11817 leader_election.cc:302] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3294537c4ba54a208a675b571a992a9d; no voters:
I20260926 07:38:32.989068 11817 leader_election.cc:288] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:32.989179 11827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:32.990860 11827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [term 1 LEADER]: Becoming Leader. State: Replica: 3294537c4ba54a208a675b571a992a9d, State: Running, Role: LEADER
I20260926 07:38:32.991556 11827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294537c4ba54a208a675b571a992a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260926 07:38:32.991904 11817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:32.995069 11828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3294537c4ba54a208a675b571a992a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294537c4ba54a208a675b571a992a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260926 07:38:32.995599 11828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:32.997247 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:32.983278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49373 (local address 127.0.0.1:39367)
0926 07:38:32.983432 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:32.983439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:32.988462 (+ 5023us) server_negotiation.cc:407] Connection header received
0926 07:38:32.988760 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:32.988769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:32.988882 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:32.989005 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:32.989731 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:32.990986 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:32.992713 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:32.993372 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:32.993520 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:32.995081 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:32.995087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:32.995092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:32.995114 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:32.995164 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:32.995187 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:32.995392 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:32.996126 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 07:38:32.996264 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:38:33.000654 11834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:33.000438 11829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3294537c4ba54a208a675b571a992a9d. Latest consensus state: current_term: 1 leader_uuid: "3294537c4ba54a208a675b571a992a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294537c4ba54a208a675b571a992a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260926 07:38:33.000857 11829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:33.006580 11834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:33.016583 11834 catalog_manager.cc:1347] Generated new cluster ID: c810916531554e0abc754d2043f50cbb
I20260926 07:38:33.016708 11834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:33.038275 11834 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:33.040185 11834 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:33.052449 11834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3294537c4ba54a208a675b571a992a9d: Generated new TSK 0
I20260926 07:38:33.053591 11834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:33.069552 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11750
W20260926 07:38:33.070699 11812 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44985
W20260926 07:38:33.071484 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11750: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:33.071636 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11750
2026-09-26T07:38:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:33 dist-test-slave-qlq8 krb5kdc[11848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:33 dist-test-slave-qlq8 krb5kdc[11848](info): set up 2 sockets
Sep 26 07:38:33 dist-test-slave-qlq8 krb5kdc[11848](info): commencing operation
krb5kdc: starting...
W20260926 07:38:34.183753 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:34 dist-test-slave-qlq8 krb5kdc[11848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:34.436553 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:34.771443 11861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:34.771817 11861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:34.771984 11861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:34.779836 11861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:34.780010 11861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:34.780117 11861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:34.780229 11861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:34.780308 11861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:34.791528 11861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:34.792904 11861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:34.794227 11861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:34.806243 11867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:34.806200 11869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:34.806295 11866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:34.807112 11861 server_base.cc:1034] running on GCE node
I20260926 07:38:34.807983 11861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:34.809350 11861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:34.810616 11861 hybrid_clock.cc:648] HybridClock initialized: now 1790408314810562 us; error 36 us; skew 500 ppm
I20260926 07:38:34.811067 11861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:34 dist-test-slave-qlq8 krb5kdc[11848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:34.814242 11861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:34.815860 11861 webserver.cc:460] Webserver started at http://127.0.0.1:43889/ using document root <none> and password file <none>
I20260926 07:38:34.816644 11861 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:34.816792 11861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:34.817096 11861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:34.820182 11861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "da82311b7345401c87703d0e13400d99"
format_stamp: "Formatted at 2026-09-26 07:38:34 on dist-test-slave-qlq8"
server_key: "06ea98d0d7796f18f884d76c2e4fee19"
server_key_iv: "b3bc3e9a65a88a8e32ada8914cd0120a"
server_key_version: "encryptionkey@0"
I20260926 07:38:34.820982 11861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "da82311b7345401c87703d0e13400d99"
format_stamp: "Formatted at 2026-09-26 07:38:34 on dist-test-slave-qlq8"
server_key: "06ea98d0d7796f18f884d76c2e4fee19"
server_key_iv: "b3bc3e9a65a88a8e32ada8914cd0120a"
server_key_version: "encryptionkey@0"
I20260926 07:38:34.826239 11861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:38:34.829802 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:34.831209 11861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:38:34.831452 11861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "da82311b7345401c87703d0e13400d99"
format_stamp: "Formatted at 2026-09-26 07:38:34 on dist-test-slave-qlq8"
server_key: "06ea98d0d7796f18f884d76c2e4fee19"
server_key_iv: "b3bc3e9a65a88a8e32ada8914cd0120a"
server_key_version: "encryptionkey@0"
I20260926 07:38:34.831655 11861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:34.837579 11861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:34.841858 11861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:34.842249 11861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:34.861093 11861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35429
I20260926 07:38:34.861178 11927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35429 every 8 connection(s)
I20260926 07:38:34.862483 11861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:34.866432 11928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:34.871140 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11861
I20260926 07:38:34.871378 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:34.871966 23413 external_mini_cluster.cc:1428] Setting key 2cc0b2fafd534532d2aefd460465c433
I20260926 07:38:34.878799 11928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99: Bootstrap starting.
I20260926 07:38:34.880395 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:34.873745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33641 (local address 127.0.0.1:35429)
0926 07:38:34.874525 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:34.874535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:34.874575 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:38:34.876908 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:34.876933 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:34.877285 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:34.877625 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:34.878749 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:34.878756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:34.878769 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:34.878787 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:34.878848 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:34.878868 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:34.879117 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:34.879318 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 07:38:34.879499 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":344,"threads_started":1}
I20260926 07:38:34.882359 11928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:34.883730 11928 log.cc:824] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:34.884047 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:34.887140 11928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99: No bootstrap required, opened a new log
I20260926 07:38:34.892455 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:34.889613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57841 (local address 127.0.0.1:35429)
0926 07:38:34.889816 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:34.889823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:34.890247 (+ 424us) server_negotiation.cc:407] Connection header received
0926 07:38:34.890404 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:34.890411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:34.890479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:34.890627 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:34.891666 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:34.891672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:34.891676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:34.891696 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:34.891752 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:34.891775 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:34.891924 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:34.892053 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 07:38:34.892121 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:38:34.897547 11928 raft_consensus.cc:348] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da82311b7345401c87703d0e13400d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260926 07:38:34.897953 11928 raft_consensus.cc:374] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:34.898034 11928 raft_consensus.cc:729] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da82311b7345401c87703d0e13400d99, State: Initialized, Role: FOLLOWER
I20260926 07:38:34.898634 11928 consensus_queue.cc:260] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da82311b7345401c87703d0e13400d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260926 07:38:34.898998 11928 raft_consensus.cc:388] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:34.899173 11928 raft_consensus.cc:482] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:34.899461 11928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:34.902034 11928 raft_consensus.cc:504] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da82311b7345401c87703d0e13400d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260926 07:38:34.902549 11928 leader_election.cc:302] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da82311b7345401c87703d0e13400d99; no voters:
I20260926 07:38:34.903154 11928 leader_election.cc:288] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:34.903352 11938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:34.904138 11938 raft_consensus.cc:686] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [term 1 LEADER]: Becoming Leader. State: Replica: da82311b7345401c87703d0e13400d99, State: Running, Role: LEADER
I20260926 07:38:34.904676 11938 consensus_queue.cc:237] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da82311b7345401c87703d0e13400d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260926 07:38:34.905666 11928 sys_catalog.cc:564] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:34.907908 11940 sys_catalog.cc:455] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da82311b7345401c87703d0e13400d99. Latest consensus state: current_term: 1 leader_uuid: "da82311b7345401c87703d0e13400d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da82311b7345401c87703d0e13400d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } } }
I20260926 07:38:34.908434 11940 sys_catalog.cc:458] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:34.909049 11939 sys_catalog.cc:455] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da82311b7345401c87703d0e13400d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da82311b7345401c87703d0e13400d99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } } }
I20260926 07:38:34.909385 11939 sys_catalog.cc:458] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:34.910643 11943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:34.916597 11943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:34.929553 11943 catalog_manager.cc:1347] Generated new cluster ID: 1a1c634329124894b5c0181bd987cd9f
I20260926 07:38:34.929690 11943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:34.949702 11943 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:34.950239 11943 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:34.963678 11943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da82311b7345401c87703d0e13400d99: Generated new TSK 0
I20260926 07:38:34.964560 11943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:34.973762 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11861
W20260926 07:38:34.975219 11923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33641
W20260926 07:38:34.976046 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:34.976210 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11861
2026-09-26T07:38:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:35 dist-test-slave-qlq8 krb5kdc[11959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:35 dist-test-slave-qlq8 krb5kdc[11959](info): set up 2 sockets
Sep 26 07:38:35 dist-test-slave-qlq8 krb5kdc[11959](info): commencing operation
krb5kdc: starting...
W20260926 07:38:36.140167 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:36 dist-test-slave-qlq8 krb5kdc[11959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:36.396785 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:36.728016 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:36.728283 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:36.728374 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:36.735625 11972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:36.735743 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:36.735797 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:36.735862 11972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:36.735908 11972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:36.745725 11972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:36.747001 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:36.748277 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:36.760429 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:36.760452 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:36.760382 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:36.761372 11972 server_base.cc:1034] running on GCE node
I20260926 07:38:36.762200 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:36.763700 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:36.764941 11972 hybrid_clock.cc:648] HybridClock initialized: now 1790408316764902 us; error 32 us; skew 500 ppm
I20260926 07:38:36.765290 11972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:36 dist-test-slave-qlq8 krb5kdc[11959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:36.768496 11972 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:36.770041 11972 webserver.cc:460] Webserver started at http://127.0.0.1:33105/ using document root <none> and password file <none>
I20260926 07:38:36.770735 11972 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:36.770888 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:36.771188 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:36.774199 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "0ded7666fbf949c2b1d29c4c2f878ab8"
format_stamp: "Formatted at 2026-09-26 07:38:36 on dist-test-slave-qlq8"
server_key: "17b8effae131c6c0ba23ac3c38bf8c7e"
server_key_iv: "d9ca68962bbdcd4ea370b13f3e7f0b5e"
server_key_version: "encryptionkey@0"
I20260926 07:38:36.775038 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "0ded7666fbf949c2b1d29c4c2f878ab8"
format_stamp: "Formatted at 2026-09-26 07:38:36 on dist-test-slave-qlq8"
server_key: "17b8effae131c6c0ba23ac3c38bf8c7e"
server_key_iv: "d9ca68962bbdcd4ea370b13f3e7f0b5e"
server_key_version: "encryptionkey@0"
I20260926 07:38:36.780602 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:38:36.784211 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:36.785615 11972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:36.785815 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "0ded7666fbf949c2b1d29c4c2f878ab8"
format_stamp: "Formatted at 2026-09-26 07:38:36 on dist-test-slave-qlq8"
server_key: "17b8effae131c6c0ba23ac3c38bf8c7e"
server_key_iv: "d9ca68962bbdcd4ea370b13f3e7f0b5e"
server_key_version: "encryptionkey@0"
I20260926 07:38:36.786021 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:36.797955 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:36.802254 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:36.802546 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:36.821113 11972 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40839
I20260926 07:38:36.821142 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40839 every 8 connection(s)
I20260926 07:38:36.822544 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:36.826324 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:36.830874 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 11972
I20260926 07:38:36.831060 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:36.831668 23413 external_mini_cluster.cc:1428] Setting key 3d92c5d0cb1becea900986161295a654
I20260926 07:38:36.838347 12039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8: Bootstrap starting.
I20260926 07:38:36.841634 12039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:36.842710 12039 log.cc:824] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:36.845597 12039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8: No bootstrap required, opened a new log
Sep 26 07:38:36 dist-test-slave-qlq8 krb5kdc[11959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:38:36.855134 12039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260926 07:38:36.855616 12039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:36.855718 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:36.833826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:51983 (local address 127.0.0.1:40839)
0926 07:38:36.834726 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:36.834738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:36.834778 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:38:36.836938 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:36.836962 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:36.837394 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:36.837791 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:36.839489 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:36.840827 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:36.842233 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:36.842991 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:36.843194 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:36.847995 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:36.848002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:36.848020 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:36.848057 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:36.851322 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:36.852297 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:36.852304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:36.852310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:36.852466 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:36.852946 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:36.852950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:36.852955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:36.853489 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:36.853731 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:36.854218 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 07:38:36.854556 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":352,"threads_started":1}
I20260926 07:38:36.855736 12039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ded7666fbf949c2b1d29c4c2f878ab8, State: Initialized, Role: FOLLOWER
I20260926 07:38:36.856354 12039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260926 07:38:36.856631 12039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:36.856786 12039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:36.856977 12039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:36.859604 12039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260926 07:38:36.859839 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:36.860178 12039 leader_election.cc:302] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ded7666fbf949c2b1d29c4c2f878ab8; no voters:
I20260926 07:38:36.860780 12039 leader_election.cc:288] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:36.860911 12044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:36.862879 12039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:36.863529 12044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [term 1 LEADER]: Becoming Leader. State: Replica: 0ded7666fbf949c2b1d29c4c2f878ab8, State: Running, Role: LEADER
I20260926 07:38:36.864084 12044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } }
I20260926 07:38:36.866950 12045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } } }
I20260926 07:38:36.867344 12045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:36.867828 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ded7666fbf949c2b1d29c4c2f878ab8. Latest consensus state: current_term: 1 leader_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ded7666fbf949c2b1d29c4c2f878ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40839 } } }
I20260926 07:38:36.868120 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:36.871845 12054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:36.876814 12054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:36.886935 12054 catalog_manager.cc:1347] Generated new cluster ID: 9a15f8c9d43b4aea991cc44344386c2e
I20260926 07:38:36.887056 12054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:36.904855 12054 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:36.906764 12054 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:36.917557 12054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ded7666fbf949c2b1d29c4c2f878ab8: Generated new TSK 0
I20260926 07:38:36.918531 12054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:36.920454 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:36.896005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:38657 (local address 127.0.0.1:40839)
0926 07:38:36.896244 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:36.896253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:36.903881 (+ 7628us) server_negotiation.cc:407] Connection header received
0926 07:38:36.904241 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:36.904247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:36.904341 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:36.904455 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:36.906438 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:36.907725 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:36.909439 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:36.910100 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:36.910242 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:36.913179 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:36.913186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:36.913191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:36.913228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:36.916943 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:36.917904 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:36.917910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:36.917914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:36.918099 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:36.918666 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:36.918671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:36.918675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:36.919082 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:36.919423 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:36.919884 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 07:38:36.920064 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:38:36.925832 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 11972
I20260926 07:38:36.927076 12034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:37.017985 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 11972
W20260926 07:38:37.035526 11957 connection.cc:535] client connection to 127.0.0.1:40839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40839): Connection reset by peer (error 104)
2026-09-26T07:38:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:37 dist-test-slave-qlq8 krb5kdc[12072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:37 dist-test-slave-qlq8 krb5kdc[12072](info): set up 2 sockets
Sep 26 07:38:37 dist-test-slave-qlq8 krb5kdc[12072](info): commencing operation
krb5kdc: starting...
W20260926 07:38:38.135912 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:38 dist-test-slave-qlq8 krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:38.398761 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:38.750161 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:38.750466 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:38.750658 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:38.758229 12085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:38.758360 12085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:38.758448 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:38.758548 12085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:38.758615 12085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:38.769124 12085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:38.770577 12085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:38.771996 12085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:38.784384 12090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:38.784336 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:38.784438 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:38.785311 12085 server_base.cc:1034] running on GCE node
I20260926 07:38:38.786126 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:38.787611 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:38.788918 12085 hybrid_clock.cc:648] HybridClock initialized: now 1790408318788838 us; error 64 us; skew 500 ppm
I20260926 07:38:38.789271 12085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:38 dist-test-slave-qlq8 krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:38.792629 12085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:38.794271 12085 webserver.cc:460] Webserver started at http://127.0.0.1:43201/ using document root <none> and password file <none>
I20260926 07:38:38.794955 12085 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:38.795104 12085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:38.795463 12085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:38.798425 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "d6e2c8e4e53746dda716c5a018bc05b2"
format_stamp: "Formatted at 2026-09-26 07:38:38 on dist-test-slave-qlq8"
server_key: "9f33eafbca69cc725fba443f0ce23309"
server_key_iv: "55b641f3b90f93d6526f83f7a51e1a5d"
server_key_version: "encryptionkey@0"
I20260926 07:38:38.799203 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "d6e2c8e4e53746dda716c5a018bc05b2"
format_stamp: "Formatted at 2026-09-26 07:38:38 on dist-test-slave-qlq8"
server_key: "9f33eafbca69cc725fba443f0ce23309"
server_key_iv: "55b641f3b90f93d6526f83f7a51e1a5d"
server_key_version: "encryptionkey@0"
I20260926 07:38:38.804365 12085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:38:38.807816 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:38.809128 12085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:38:38.809315 12085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "d6e2c8e4e53746dda716c5a018bc05b2"
format_stamp: "Formatted at 2026-09-26 07:38:38 on dist-test-slave-qlq8"
server_key: "9f33eafbca69cc725fba443f0ce23309"
server_key_iv: "55b641f3b90f93d6526f83f7a51e1a5d"
server_key_version: "encryptionkey@0"
I20260926 07:38:38.809517 12085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:38.829483 12085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:38.834028 12085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:38.834319 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:38.853148 12085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35781
I20260926 07:38:38.853183 12151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35781 every 8 connection(s)
I20260926 07:38:38.854538 12085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:38.855137 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12085
I20260926 07:38:38.855353 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:38.856010 23413 external_mini_cluster.cc:1428] Setting key b519c0d1e043e65875906e1526c81923
I20260926 07:38:38.859083 12152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:38.868752 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:38.857780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:37083 (local address 127.0.0.1:35781)
0926 07:38:38.858513 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:38.858524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:38.858563 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:38:38.860225 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:38.860247 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:38.860616 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:38.860982 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:38.861529 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:38.862834 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:38.864210 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:38.864892 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:38.865089 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:38.866442 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:38.866450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:38.866465 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:38.867015 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:38.867093 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:38.867121 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:38.867329 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:38.867623 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 07:38:38.867791 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260926 07:38:38.870406 12152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2: Bootstrap starting.
I20260926 07:38:38.872886 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:38.873835 12152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:38.874854 12152 log.cc:824] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:38.877941 12152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2: No bootstrap required, opened a new log
I20260926 07:38:38.883622 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:38.877399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:54183 (local address 127.0.0.1:35781)
0926 07:38:38.877591 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:38.877599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:38.877993 (+ 394us) server_negotiation.cc:407] Connection header received
0926 07:38:38.878239 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:38.878243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:38.878311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:38.878433 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:38.878943 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:38.879800 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:38.881112 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:38.881558 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:38.881692 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:38.882839 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:38.882847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:38.882850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:38.882869 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:38.882908 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:38.882924 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:38.883059 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:38.883284 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 07:38:38.883345 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:38:38.886104 12152 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260926 07:38:38.886615 12152 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:38.886750 12152 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2c8e4e53746dda716c5a018bc05b2, State: Initialized, Role: FOLLOWER
I20260926 07:38:38.888260 12152 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260926 07:38:38.888538 12152 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:38.888720 12152 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:38.888891 12152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:38.891207 12152 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260926 07:38:38.891847 12152 leader_election.cc:302] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6e2c8e4e53746dda716c5a018bc05b2; no voters:
I20260926 07:38:38.892490 12152 leader_election.cc:288] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:38.892648 12162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:38.893365 12162 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2c8e4e53746dda716c5a018bc05b2, State: Running, Role: LEADER
I20260926 07:38:38.894065 12162 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260926 07:38:38.894851 12152 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:38.897001 12164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6e2c8e4e53746dda716c5a018bc05b2. Latest consensus state: current_term: 1 leader_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } } }
I20260926 07:38:38.897032 12163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e2c8e4e53746dda716c5a018bc05b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } } }
I20260926 07:38:38.897471 12163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:38.897471 12164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:38.899430 12167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:38.905189 12167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:38.914846 12167 catalog_manager.cc:1347] Generated new cluster ID: 689426ba62a142eb99ac2c70c8e7caf5
I20260926 07:38:38.914961 12167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:38.937652 12167 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:38.939337 12167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:38.945078 12167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6e2c8e4e53746dda716c5a018bc05b2: Generated new TSK 0
I20260926 07:38:38.945936 12167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:38.952554 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12085
W20260926 07:38:38.953946 12147 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:37083
W20260926 07:38:38.954774 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12085: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:38.954943 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12085
W20260926 07:38:38.971626 12070 connection.cc:535] client connection to 127.0.0.1:35781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35781): Connection reset by peer (error 104)
2026-09-26T07:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:39 dist-test-slave-qlq8 krb5kdc[12183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:39 dist-test-slave-qlq8 krb5kdc[12183](info): set up 2 sockets
Sep 26 07:38:39 dist-test-slave-qlq8 krb5kdc[12183](info): commencing operation
krb5kdc: starting...
W20260926 07:38:40.072461 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:40 dist-test-slave-qlq8 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:40.351006 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:40.682432 12196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:40.682706 12196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:40.682894 12196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:40.690061 12196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:40.690191 12196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:40.690299 12196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:40.690397 12196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:40.690480 12196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:40.700798 12196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:40.702072 12196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:40.703378 12196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:40.715420 12201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:40.715416 12202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:40.715443 12204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:40.716318 12196 server_base.cc:1034] running on GCE node
I20260926 07:38:40.717060 12196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:40.718463 12196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:40.719683 12196 hybrid_clock.cc:648] HybridClock initialized: now 1790408320719638 us; error 40 us; skew 500 ppm
I20260926 07:38:40.720037 12196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:40 dist-test-slave-qlq8 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:40.723019 12196 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:40.724615 12196 webserver.cc:460] Webserver started at http://127.0.0.1:34031/ using document root <none> and password file <none>
I20260926 07:38:40.725286 12196 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:40.725427 12196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:40.725723 12196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:40.728736 12196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "519488e1052a48b6921e5aa75cb64136"
format_stamp: "Formatted at 2026-09-26 07:38:40 on dist-test-slave-qlq8"
server_key: "cfd9c43d30e2befe46269065807e48ef"
server_key_iv: "079ef5a85017e2c8ce4c467516d93f22"
server_key_version: "encryptionkey@0"
I20260926 07:38:40.729533 12196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "519488e1052a48b6921e5aa75cb64136"
format_stamp: "Formatted at 2026-09-26 07:38:40 on dist-test-slave-qlq8"
server_key: "cfd9c43d30e2befe46269065807e48ef"
server_key_iv: "079ef5a85017e2c8ce4c467516d93f22"
server_key_version: "encryptionkey@0"
I20260926 07:38:40.735083 12196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:38:40.738797 12211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:40.740520 12196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:40.740737 12196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "519488e1052a48b6921e5aa75cb64136"
format_stamp: "Formatted at 2026-09-26 07:38:40 on dist-test-slave-qlq8"
server_key: "cfd9c43d30e2befe46269065807e48ef"
server_key_iv: "079ef5a85017e2c8ce4c467516d93f22"
server_key_version: "encryptionkey@0"
I20260926 07:38:40.741014 12196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:40.747222 12196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:40.751585 12196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:40.751901 12196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:40.770790 12196 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36081
I20260926 07:38:40.770828 12262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36081 every 8 connection(s)
I20260926 07:38:40.772228 12196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:40.775295 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12196
I20260926 07:38:40.775480 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:40.776024 23413 external_mini_cluster.cc:1428] Setting key e5f3ee171ac894d46c0cba4faa5462c5
I20260926 07:38:40.776454 12263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:40.784457 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:40.777859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:52469 (local address 127.0.0.1:36081)
0926 07:38:40.778661 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:40.778673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:40.778713 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:38:40.781044 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:40.781066 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:40.781413 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:40.781744 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:40.782837 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:40.782843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:40.782856 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:40.782869 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:40.782928 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:40.782950 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:40.783118 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:40.783322 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:38:40.783571 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":373,"threads_started":1}
I20260926 07:38:40.787988 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:40.790067 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136: Bootstrap starting.
I20260926 07:38:40.794080 12263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:40.795472 12263 log.cc:824] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:40.796114 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:40.793518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:57923 (local address 127.0.0.1:36081)
0926 07:38:40.793790 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:40.793795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:40.794102 (+ 307us) server_negotiation.cc:407] Connection header received
0926 07:38:40.794341 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:40.794345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:40.794395 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:40.794480 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:40.795446 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:40.795453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:40.795456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:40.795470 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:40.795505 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:40.795518 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:40.795639 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:40.795737 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:38:40.795815 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:38:40.798702 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136: No bootstrap required, opened a new log
I20260926 07:38:40.805804 12263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519488e1052a48b6921e5aa75cb64136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260926 07:38:40.806167 12263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:40.806272 12263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519488e1052a48b6921e5aa75cb64136, State: Initialized, Role: FOLLOWER
I20260926 07:38:40.806917 12263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519488e1052a48b6921e5aa75cb64136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260926 07:38:40.807153 12263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:40.807309 12263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:40.807499 12263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:40.809576 12263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519488e1052a48b6921e5aa75cb64136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260926 07:38:40.810063 12263 leader_election.cc:302] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519488e1052a48b6921e5aa75cb64136; no voters:
I20260926 07:38:40.810639 12263 leader_election.cc:288] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:40.810810 12273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:40.811486 12273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [term 1 LEADER]: Becoming Leader. State: Replica: 519488e1052a48b6921e5aa75cb64136, State: Running, Role: LEADER
I20260926 07:38:40.811993 12273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519488e1052a48b6921e5aa75cb64136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } }
I20260926 07:38:40.812654 12263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:40.814497 12274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519488e1052a48b6921e5aa75cb64136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519488e1052a48b6921e5aa75cb64136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } } }
I20260926 07:38:40.815047 12274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:40.814867 12275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519488e1052a48b6921e5aa75cb64136. Latest consensus state: current_term: 1 leader_uuid: "519488e1052a48b6921e5aa75cb64136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519488e1052a48b6921e5aa75cb64136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36081 } } }
I20260926 07:38:40.815331 12275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:40.816936 12279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:40.822856 12279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:40.832140 12279 catalog_manager.cc:1347] Generated new cluster ID: 8e49492c6e89486a9f05d872a9b335cf
I20260926 07:38:40.832261 12279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:40.854494 12279 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:40.854947 12279 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:40.865528 12279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519488e1052a48b6921e5aa75cb64136: Generated new TSK 0
I20260926 07:38:40.866389 12279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:40.875969 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12196
W20260926 07:38:40.877089 12258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:52469
W20260926 07:38:40.877827 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:40.877997 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12196
2026-09-26T07:38:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:40 dist-test-slave-qlq8 krb5kdc[12294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:40 dist-test-slave-qlq8 krb5kdc[12294](info): set up 2 sockets
Sep 26 07:38:40 dist-test-slave-qlq8 krb5kdc[12294](info): commencing operation
krb5kdc: starting...
W20260926 07:38:41.983883 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:42 dist-test-slave-qlq8 krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:42.254662 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:42.595430 12307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:42.595748 12307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:42.595952 12307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:42.603168 12307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:42.603394 12307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:42.603505 12307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:42.603617 12307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:42.603703 12307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:42.613739 12307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:42.615127 12307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:42.616492 12307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:42.628933 12313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:42.629032 12312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:42.629073 12315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:42.629755 12307 server_base.cc:1034] running on GCE node
I20260926 07:38:42.630586 12307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:42.632141 12307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:42.633368 12307 hybrid_clock.cc:648] HybridClock initialized: now 1790408322633321 us; error 32 us; skew 500 ppm
I20260926 07:38:42.633742 12307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:42 dist-test-slave-qlq8 krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:42.637041 12307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:42.638568 12307 webserver.cc:460] Webserver started at http://127.0.0.1:36067/ using document root <none> and password file <none>
I20260926 07:38:42.639351 12307 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:42.639501 12307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:42.639815 12307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:42.642942 12307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "3a0b473c95a54d85a8e0622f0e443ddb"
format_stamp: "Formatted at 2026-09-26 07:38:42 on dist-test-slave-qlq8"
server_key: "cb65388ef3bec0f232525f78ff7de6d6"
server_key_iv: "89b639d2a6e1f809b9ade3eec4106767"
server_key_version: "encryptionkey@0"
I20260926 07:38:42.643793 12307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "3a0b473c95a54d85a8e0622f0e443ddb"
format_stamp: "Formatted at 2026-09-26 07:38:42 on dist-test-slave-qlq8"
server_key: "cb65388ef3bec0f232525f78ff7de6d6"
server_key_iv: "89b639d2a6e1f809b9ade3eec4106767"
server_key_version: "encryptionkey@0"
I20260926 07:38:42.649158 12307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:38:42.652680 12322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:42.654081 12307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:42.654270 12307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "3a0b473c95a54d85a8e0622f0e443ddb"
format_stamp: "Formatted at 2026-09-26 07:38:42 on dist-test-slave-qlq8"
server_key: "cb65388ef3bec0f232525f78ff7de6d6"
server_key_iv: "89b639d2a6e1f809b9ade3eec4106767"
server_key_version: "encryptionkey@0"
I20260926 07:38:42.654472 12307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:42.669042 12307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:42.673043 12307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:42.673343 12307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:42.691658 12307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40495
I20260926 07:38:42.691689 12373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40495 every 8 connection(s)
I20260926 07:38:42.693018 12307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:42.696682 12374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:42.700137 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12307
I20260926 07:38:42.700358 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:42.700906 23413 external_mini_cluster.cc:1428] Setting key e14f12a4d994ead818787552d557ccfc
I20260926 07:38:42.709647 12374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb: Bootstrap starting.
I20260926 07:38:42.713112 12374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:42.714289 12374 log.cc:824] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:38:42 dist-test-slave-qlq8 krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:38:42.717643 12374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb: No bootstrap required, opened a new log
I20260926 07:38:42.725981 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:42.702957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:59857 (local address 127.0.0.1:40495)
0926 07:38:42.703739 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:42.703751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:42.703833 (+ 82us) server_negotiation.cc:407] Connection header received
0926 07:38:42.706044 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:42.706079 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:42.706547 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:42.706989 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:42.708614 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:42.710012 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:42.711470 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:42.712185 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:42.712401 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:42.717036 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:42.717048 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:42.717078 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:42.717135 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:42.721191 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:42.722055 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:42.722062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:42.722081 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:42.722304 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:42.722739 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:42.722744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:42.722747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:42.723498 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:42.723868 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:42.724201 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:38:42.724564 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260926 07:38:42.728534 12374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260926 07:38:42.729007 12374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:42.729120 12374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0b473c95a54d85a8e0622f0e443ddb, State: Initialized, Role: FOLLOWER
I20260926 07:38:42.729759 12374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260926 07:38:42.730026 12374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:42.730167 12374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:42.730333 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:42.730358 12374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:42.733621 12374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260926 07:38:42.734305 12374 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a0b473c95a54d85a8e0622f0e443ddb; no voters:
I20260926 07:38:42.735045 12374 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:42.735163 12383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:42.737349 12374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:42.741801 12383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [term 1 LEADER]: Becoming Leader. State: Replica: 3a0b473c95a54d85a8e0622f0e443ddb, State: Running, Role: LEADER
I20260926 07:38:42.742513 12383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260926 07:38:42.746042 12384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } } }
I20260926 07:38:42.746440 12384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:42.746848 12385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0b473c95a54d85a8e0622f0e443ddb. Latest consensus state: current_term: 1 leader_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0b473c95a54d85a8e0622f0e443ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } } }
I20260926 07:38:42.747818 12385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:42.752501 12394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:42.757104 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:42.736569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:60335 (local address 127.0.0.1:40495)
0926 07:38:42.736737 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:42.736744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:42.736876 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:38:42.737224 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:42.737234 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:42.737341 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:42.737428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:42.740170 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:42.741728 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:42.743520 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:42.744351 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:42.744489 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:42.748143 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:42.748150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:42.748158 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:42.748208 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:42.752751 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:42.753701 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:42.753706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:42.753709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:42.753878 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:42.754468 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:42.754474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:42.754477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:42.755019 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:42.755390 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:42.756313 (+ 923us) server_negotiation.cc:299] Negotiation successful
0926 07:38:42.756568 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:38:42.762097 12394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:42.770826 12394 catalog_manager.cc:1347] Generated new cluster ID: 35ef8c9bffff4fd7adac1007f7108d75
I20260926 07:38:42.770944 12394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:42.789060 12394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:42.790403 12394 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:42.799501 12394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a0b473c95a54d85a8e0622f0e443ddb: Generated new TSK 0
I20260926 07:38:42.800618 12394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:42.812258 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12307
I20260926 07:38:42.813575 12369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:42.907330 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12307
W20260926 07:38:42.921718 12292 connection.cc:535] client connection to 127.0.0.1:40495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40495): Connection reset by peer (error 104)
2026-09-26T07:38:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:43 dist-test-slave-qlq8 krb5kdc[12407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:43 dist-test-slave-qlq8 krb5kdc[12407](info): set up 2 sockets
Sep 26 07:38:43 dist-test-slave-qlq8 krb5kdc[12407](info): commencing operation
krb5kdc: starting...
W20260926 07:38:44.019743 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:44 dist-test-slave-qlq8 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:44.284282 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:44.625278 12420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:44.625659 12420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:44.625828 12420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:44.633052 12420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:44.633214 12420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:44.633327 12420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:44.633441 12420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:44.633538 12420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:44.643574 12420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:44.644897 12420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:44.646198 12420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:44.657998 12425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:44.657990 12426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:44.658506 12428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:44.658764 12420 server_base.cc:1034] running on GCE node
I20260926 07:38:44.659577 12420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:44.660902 12420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:44.662133 12420 hybrid_clock.cc:648] HybridClock initialized: now 1790408324662067 us; error 52 us; skew 500 ppm
I20260926 07:38:44.662513 12420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:44 dist-test-slave-qlq8 krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:44.668767 12420 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:44.670177 12420 webserver.cc:460] Webserver started at http://127.0.0.1:35749/ using document root <none> and password file <none>
I20260926 07:38:44.670816 12420 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:44.670914 12420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:44.671166 12420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:44.674081 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "3280213fba2841cc917a6027ebb9c867"
format_stamp: "Formatted at 2026-09-26 07:38:44 on dist-test-slave-qlq8"
server_key: "f8804ec3753e313885f6494643810d83"
server_key_iv: "2e7c0aa1b07354a23772530bca1999b8"
server_key_version: "encryptionkey@0"
I20260926 07:38:44.674772 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "3280213fba2841cc917a6027ebb9c867"
format_stamp: "Formatted at 2026-09-26 07:38:44 on dist-test-slave-qlq8"
server_key: "f8804ec3753e313885f6494643810d83"
server_key_iv: "2e7c0aa1b07354a23772530bca1999b8"
server_key_version: "encryptionkey@0"
I20260926 07:38:44.679853 12420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:38:44.683400 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:44.684953 12420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:38:44.685159 12420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "3280213fba2841cc917a6027ebb9c867"
format_stamp: "Formatted at 2026-09-26 07:38:44 on dist-test-slave-qlq8"
server_key: "f8804ec3753e313885f6494643810d83"
server_key_iv: "2e7c0aa1b07354a23772530bca1999b8"
server_key_version: "encryptionkey@0"
I20260926 07:38:44.685360 12420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:44.720636 12420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:44.726331 12420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:44.726723 12420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:44.747206 12420 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33473
I20260926 07:38:44.747346 12486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33473 every 8 connection(s)
I20260926 07:38:44.748688 12420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:44.749706 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12420
I20260926 07:38:44.749894 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:44.750445 23413 external_mini_cluster.cc:1428] Setting key d2aa64e95f141b12afdc636c69ab27a9
I20260926 07:38:44.753410 12487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:44.764480 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:44.752397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:33645 (local address 127.0.0.1:33473)
0926 07:38:44.753189 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:44.753200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:44.753241 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:38:44.754954 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:44.754977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:44.755457 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:44.755795 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:44.756445 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:44.757795 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:44.759689 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:44.760390 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:44.760546 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:44.761936 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:44.761943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:44.761957 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:44.762527 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:44.762641 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:44.762669 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:44.762808 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:44.763150 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 07:38:44.763385 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":365,"threads_started":1}
I20260926 07:38:44.766688 12487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867: Bootstrap starting.
I20260926 07:38:44.768841 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:44.770206 12487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:44.771306 12487 log.cc:824] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:44.774683 12487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867: No bootstrap required, opened a new log
I20260926 07:38:44.781723 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:44.773272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:41147 (local address 127.0.0.1:33473)
0926 07:38:44.773446 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:44.773451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:44.773968 (+ 517us) server_negotiation.cc:407] Connection header received
0926 07:38:44.774123 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:44.774137 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:44.774218 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:44.774344 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:44.774944 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:44.776116 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:44.778103 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:44.778975 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:44.779169 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:44.780682 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:44.780687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:44.780691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:44.780710 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:44.780769 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:44.780785 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:44.780940 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:44.781216 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:38:44.781273 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:38:44.786940 12487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3280213fba2841cc917a6027ebb9c867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260926 07:38:44.787389 12487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:44.787523 12487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3280213fba2841cc917a6027ebb9c867, State: Initialized, Role: FOLLOWER
I20260926 07:38:44.788185 12487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3280213fba2841cc917a6027ebb9c867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260926 07:38:44.788506 12487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:44.788676 12487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:44.788873 12487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:44.791417 12487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3280213fba2841cc917a6027ebb9c867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260926 07:38:44.792024 12487 leader_election.cc:302] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3280213fba2841cc917a6027ebb9c867; no voters:
I20260926 07:38:44.792776 12487 leader_election.cc:288] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:44.792936 12497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:44.793644 12497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [term 1 LEADER]: Becoming Leader. State: Replica: 3280213fba2841cc917a6027ebb9c867, State: Running, Role: LEADER
I20260926 07:38:44.794206 12497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3280213fba2841cc917a6027ebb9c867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260926 07:38:44.795009 12487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:44.797055 12498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3280213fba2841cc917a6027ebb9c867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3280213fba2841cc917a6027ebb9c867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } } }
I20260926 07:38:44.797468 12498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:44.797844 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3280213fba2841cc917a6027ebb9c867. Latest consensus state: current_term: 1 leader_uuid: "3280213fba2841cc917a6027ebb9c867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3280213fba2841cc917a6027ebb9c867" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } } }
I20260926 07:38:44.798122 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:44.801815 12502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:44.807176 12502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:44.816071 12502 catalog_manager.cc:1347] Generated new cluster ID: c9a5466230a6405faaaab4786ab4f79f
I20260926 07:38:44.816196 12502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:44.826859 12502 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:44.828266 12502 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:44.837921 12502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3280213fba2841cc917a6027ebb9c867: Generated new TSK 0
I20260926 07:38:44.838765 12502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:44.850757 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12420
W20260926 07:38:44.852240 12482 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:33645
W20260926 07:38:44.853008 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12420: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:44.853142 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12420
W20260926 07:38:44.869727 12405 connection.cc:535] client connection to 127.0.0.1:33473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33473): Connection reset by peer (error 104)
2026-09-26T07:38:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:44 dist-test-slave-qlq8 krb5kdc[12518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:44 dist-test-slave-qlq8 krb5kdc[12518](info): set up 2 sockets
Sep 26 07:38:44 dist-test-slave-qlq8 krb5kdc[12518](info): commencing operation
krb5kdc: starting...
W20260926 07:38:45.963825 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:46 dist-test-slave-qlq8 krb5kdc[12518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:38:46.232934 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:46.572790 12531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:46.573191 12531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:46.573364 12531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:46.580895 12531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:46.581051 12531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:46.581171 12531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:46.581275 12531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:46.581362 12531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:46.591724 12531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:46.593138 12531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:46.594520 12531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:46.607031 12537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:46.607376 12539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:46.607430 12536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:46.608405 12531 server_base.cc:1034] running on GCE node
I20260926 07:38:46.609292 12531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:46.610821 12531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:46.612072 12531 hybrid_clock.cc:648] HybridClock initialized: now 1790408326612024 us; error 39 us; skew 500 ppm
I20260926 07:38:46.612452 12531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:46 dist-test-slave-qlq8 krb5kdc[12518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:46.615762 12531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:38:46.617420 12531 webserver.cc:460] Webserver started at http://127.0.0.1:40529/ using document root <none> and password file <none>
I20260926 07:38:46.618232 12531 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:46.618381 12531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:46.618691 12531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:46.621802 12531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "afc4f9df4fac41769b298abdc5d7f979"
format_stamp: "Formatted at 2026-09-26 07:38:46 on dist-test-slave-qlq8"
server_key: "6f89a649ab5ba07985d2c0d461582d39"
server_key_iv: "64110092b23bccb88c2069a94d228acd"
server_key_version: "encryptionkey@0"
I20260926 07:38:46.622599 12531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "afc4f9df4fac41769b298abdc5d7f979"
format_stamp: "Formatted at 2026-09-26 07:38:46 on dist-test-slave-qlq8"
server_key: "6f89a649ab5ba07985d2c0d461582d39"
server_key_iv: "64110092b23bccb88c2069a94d228acd"
server_key_version: "encryptionkey@0"
I20260926 07:38:46.627924 12531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 07:38:46.631481 12546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:46.632797 12531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:38:46.632980 12531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "afc4f9df4fac41769b298abdc5d7f979"
format_stamp: "Formatted at 2026-09-26 07:38:46 on dist-test-slave-qlq8"
server_key: "6f89a649ab5ba07985d2c0d461582d39"
server_key_iv: "64110092b23bccb88c2069a94d228acd"
server_key_version: "encryptionkey@0"
I20260926 07:38:46.633194 12531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:46.638978 12531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:46.643491 12531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:46.643803 12531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:46.662791 12531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33611
I20260926 07:38:46.662873 12597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33611 every 8 connection(s)
I20260926 07:38:46.664286 12531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:46.668081 12598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:46.668984 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12531
I20260926 07:38:46.669160 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:46.669740 23413 external_mini_cluster.cc:1428] Setting key 45a38c6381718a53aff8eafe4b720713
I20260926 07:38:46.677821 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:46.671622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:46869 (local address 127.0.0.1:33611)
0926 07:38:46.672421 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:46.672432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:46.672478 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:38:46.674313 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:46.674339 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:46.674694 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:46.675031 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:46.676197 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:46.676204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:46.676218 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:46.676234 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:46.676298 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:46.676322 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:46.676503 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:46.676628 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:38:46.676866 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":350,"threads_started":1}
I20260926 07:38:46.681351 12598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979: Bootstrap starting.
I20260926 07:38:46.681681 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:46.684484 12598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:46.685470 12598 log.cc:824] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:46.688606 12598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979: No bootstrap required, opened a new log
I20260926 07:38:46.688908 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:46.686131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:51193 (local address 127.0.0.1:33611)
0926 07:38:46.686310 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:46.686316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:46.686848 (+ 532us) server_negotiation.cc:407] Connection header received
0926 07:38:46.687052 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:46.687062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:46.687107 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:46.687218 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:46.688180 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:46.688193 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:46.688207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:46.688221 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:46.688257 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:46.688272 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:46.688396 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:46.688492 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:38:46.688554 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:38:46.696633 12598 raft_consensus.cc:348] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc4f9df4fac41769b298abdc5d7f979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260926 07:38:46.697074 12598 raft_consensus.cc:374] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:46.697209 12598 raft_consensus.cc:729] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc4f9df4fac41769b298abdc5d7f979, State: Initialized, Role: FOLLOWER
I20260926 07:38:46.697834 12598 consensus_queue.cc:260] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc4f9df4fac41769b298abdc5d7f979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260926 07:38:46.698055 12598 raft_consensus.cc:388] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:46.698215 12598 raft_consensus.cc:482] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:46.698403 12598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:46.700686 12598 raft_consensus.cc:504] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc4f9df4fac41769b298abdc5d7f979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260926 07:38:46.701195 12598 leader_election.cc:302] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afc4f9df4fac41769b298abdc5d7f979; no voters:
I20260926 07:38:46.701824 12598 leader_election.cc:288] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:46.701942 12608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:46.703054 12608 raft_consensus.cc:686] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [term 1 LEADER]: Becoming Leader. State: Replica: afc4f9df4fac41769b298abdc5d7f979, State: Running, Role: LEADER
I20260926 07:38:46.703722 12608 consensus_queue.cc:237] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc4f9df4fac41769b298abdc5d7f979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260926 07:38:46.703981 12598 sys_catalog.cc:564] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:46.706772 12610 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afc4f9df4fac41769b298abdc5d7f979. Latest consensus state: current_term: 1 leader_uuid: "afc4f9df4fac41769b298abdc5d7f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc4f9df4fac41769b298abdc5d7f979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } } }
I20260926 07:38:46.707417 12609 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afc4f9df4fac41769b298abdc5d7f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc4f9df4fac41769b298abdc5d7f979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } } }
I20260926 07:38:46.707692 12609 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:46.707193 12610 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:46.709760 12616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:46.715317 12616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:46.724956 12616 catalog_manager.cc:1347] Generated new cluster ID: a8d30a6d9a524a14a211fbe5da22f6f4
I20260926 07:38:46.725198 12616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:46.750598 12616 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:46.751050 12616 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:46.765895 12616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afc4f9df4fac41769b298abdc5d7f979: Generated new TSK 0
I20260926 07:38:46.766764 12616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:46.781424 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12531
W20260926 07:38:46.782930 12593 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:46869
W20260926 07:38:46.783777 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12531: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:46.783929 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12531
2026-09-26T07:38:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:46 dist-test-slave-qlq8 krb5kdc[12629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:46 dist-test-slave-qlq8 krb5kdc[12629](info): set up 2 sockets
Sep 26 07:38:46 dist-test-slave-qlq8 krb5kdc[12629](info): commencing operation
krb5kdc: starting...
W20260926 07:38:47.943894 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:48 dist-test-slave-qlq8 krb5kdc[12629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:48.210891 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43019
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:48.565768 12642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:48.566074 12642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:48.566272 12642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:48.574057 12642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:48.574177 12642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:48.574271 12642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:48.574378 12642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:48.574462 12642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:48.584947 12642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:43019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:48.586316 12642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:48.587769 12642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:48.599880 12647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:48.599815 12648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:48.599852 12650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:48.601162 12642 server_base.cc:1034] running on GCE node
I20260926 07:38:48.601963 12642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:48.603487 12642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:48.604688 12642 hybrid_clock.cc:648] HybridClock initialized: now 1790408328604623 us; error 61 us; skew 500 ppm
I20260926 07:38:48.604998 12642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:48 dist-test-slave-qlq8 krb5kdc[12629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408328, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:48.608235 12642 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:48.609772 12642 webserver.cc:460] Webserver started at http://127.22.221.126:41601/ using document root <none> and password file <none>
I20260926 07:38:48.610589 12642 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:48.610762 12642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:48.611171 12642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:48.614269 12642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "1d4b281b3fba4db0838e68916e034502"
format_stamp: "Formatted at 2026-09-26 07:38:48 on dist-test-slave-qlq8"
server_key: "1cb36b90c442c1467fbc41536727c736"
server_key_iv: "d2ec3b152208a6210833bf91eaee78aa"
server_key_version: "encryptionkey@0"
I20260926 07:38:48.615103 12642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "1d4b281b3fba4db0838e68916e034502"
format_stamp: "Formatted at 2026-09-26 07:38:48 on dist-test-slave-qlq8"
server_key: "1cb36b90c442c1467fbc41536727c736"
server_key_iv: "d2ec3b152208a6210833bf91eaee78aa"
server_key_version: "encryptionkey@0"
I20260926 07:38:48.620519 12642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 07:38:48.624270 12657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:48.625701 12642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:38:48.625902 12642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "1d4b281b3fba4db0838e68916e034502"
format_stamp: "Formatted at 2026-09-26 07:38:48 on dist-test-slave-qlq8"
server_key: "1cb36b90c442c1467fbc41536727c736"
server_key_iv: "d2ec3b152208a6210833bf91eaee78aa"
server_key_version: "encryptionkey@0"
I20260926 07:38:48.626107 12642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:48.650008 12642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:48.654649 12642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:48.655069 12642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:48.674862 12642 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:43019
I20260926 07:38:48.674892 12708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:43019 every 8 connection(s)
I20260926 07:38:48.676429 12642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:48.680572 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:48.686165 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12642
I20260926 07:38:48.686379 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:48.686823 23413 external_mini_cluster.cc:1428] Setting key 369941baee68eb6c55966b794d0ded1c
I20260926 07:38:48.693059 12709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502: Bootstrap starting.
I20260926 07:38:48.696969 12709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:48.698071 12709 log.cc:824] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:48.701167 12709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502: No bootstrap required, opened a new log
Sep 26 07:38:48 dist-test-slave-qlq8 krb5kdc[12629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:48.711974 12709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b281b3fba4db0838e68916e034502" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43019 } }
I20260926 07:38:48.712330 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:48.688769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36097 (local address 127.22.221.126:43019)
0926 07:38:48.689659 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:48.689675 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:48.689732 (+ 57us) server_negotiation.cc:407] Connection header received
0926 07:38:48.692153 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:48.692182 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:48.692719 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:48.693197 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:48.694602 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:48.696684 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:48.698107 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:48.699094 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:48.699306 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:48.703371 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:48.703381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:48.703407 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:48.703467 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:48.707685 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:48.708506 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:48.708514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:48.708523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:48.708746 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:48.709165 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:48.709172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:48.709179 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:48.709892 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:48.710249 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:48.710600 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:38:48.710965 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":405,"threads_started":1}
I20260926 07:38:48.712436 12709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:48.712544 12709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4b281b3fba4db0838e68916e034502, State: Initialized, Role: FOLLOWER
I20260926 07:38:48.713239 12709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b281b3fba4db0838e68916e034502" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43019 } }
I20260926 07:38:48.713510 12709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:48.713677 12709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:48.713876 12709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:48.716356 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:48.716570 12709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b281b3fba4db0838e68916e034502" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43019 } }
I20260926 07:38:48.717070 12709 leader_election.cc:302] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d4b281b3fba4db0838e68916e034502; no voters:
I20260926 07:38:48.717662 12709 leader_election.cc:288] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:48.717818 12714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:48.719799 12709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:48.727856 12714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [term 1 LEADER]: Becoming Leader. State: Replica: 1d4b281b3fba4db0838e68916e034502, State: Running, Role: LEADER
I20260926 07:38:48.728785 12714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b281b3fba4db0838e68916e034502" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43019 } }
I20260926 07:38:48.733691 12716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d4b281b3fba4db0838e68916e034502. Latest consensus state: current_term: 1 leader_uuid: "1d4b281b3fba4db0838e68916e034502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b281b3fba4db0838e68916e034502" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43019 } } }
I20260926 07:38:48.733727 12715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d4b281b3fba4db0838e68916e034502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4b281b3fba4db0838e68916e034502" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 43019 } } }
I20260926 07:38:48.734105 12716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:48.734107 12715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:48.745606 12730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:48.750758 12730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:48.755201 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:48.731814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49405 (local address 127.22.221.126:43019)
0926 07:38:48.732047 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:48.732056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:48.735255 (+ 3199us) server_negotiation.cc:407] Connection header received
0926 07:38:48.735584 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:48.735593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:48.735692 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:48.735780 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:48.738492 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:48.740233 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:48.742767 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:48.743755 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:48.743886 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:48.747549 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:48.747554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:48.747557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:48.747590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:48.751705 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:48.752669 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:48.752675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:48.752679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:48.752841 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:48.753375 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:48.753381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:48.753388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:48.753890 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:48.754205 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:48.754619 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 07:38:48.754826 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:38:48.760653 12730 catalog_manager.cc:1347] Generated new cluster ID: f01f13114c3147059130e5d13bb5fd5c
I20260926 07:38:48.760767 12730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:48.777305 12730 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:48.778868 12730 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:48.787122 12730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d4b281b3fba4db0838e68916e034502: Generated new TSK 0
I20260926 07:38:48.788194 12730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:48.798761 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12642
I20260926 07:38:48.800345 12704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:48.891185 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12642
W20260926 07:38:48.908490 12627 connection.cc:535] client connection to 127.22.221.126:43019 recv error: Network error: failed to read from TLS socket (remote: 127.22.221.126:43019): Connection reset by peer (error 104)
2026-09-26T07:38:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:48 dist-test-slave-qlq8 krb5kdc[12741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:48 dist-test-slave-qlq8 krb5kdc[12741](info): set up 2 sockets
Sep 26 07:38:48 dist-test-slave-qlq8 krb5kdc[12741](info): commencing operation
krb5kdc: starting...
W20260926 07:38:49.995810 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:50 dist-test-slave-qlq8 krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:50.256655 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37499
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:50.591755 12754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:50.592053 12754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:50.592268 12754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:50.599615 12754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:50.599743 12754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:50.599850 12754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:50.599953 12754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:50.600047 12754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:50.609788 12754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:37499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:50.611047 12754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:50.612351 12754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:50.624325 12759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:50.624286 12762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:50.624367 12760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:50.625135 12754 server_base.cc:1034] running on GCE node
I20260926 07:38:50.625902 12754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:50.627364 12754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:50.628576 12754 hybrid_clock.cc:648] HybridClock initialized: now 1790408330628515 us; error 59 us; skew 500 ppm
I20260926 07:38:50.628922 12754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:50 dist-test-slave-qlq8 krb5kdc[12741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408330, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:50.638882 12754 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:50.640542 12754 webserver.cc:460] Webserver started at http://127.22.221.126:37905/ using document root <none> and password file <none>
I20260926 07:38:50.641196 12754 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:50.641295 12754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:50.641539 12754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:50.644496 12754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "d43313e77842451dba471ff89e02758d"
format_stamp: "Formatted at 2026-09-26 07:38:50 on dist-test-slave-qlq8"
server_key: "a8bd2dafea88172b5ad9768dd51f8a9b"
server_key_iv: "05fedc9aafab85d566846ed0545e53bb"
server_key_version: "encryptionkey@0"
I20260926 07:38:50.645208 12754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "d43313e77842451dba471ff89e02758d"
format_stamp: "Formatted at 2026-09-26 07:38:50 on dist-test-slave-qlq8"
server_key: "a8bd2dafea88172b5ad9768dd51f8a9b"
server_key_iv: "05fedc9aafab85d566846ed0545e53bb"
server_key_version: "encryptionkey@0"
I20260926 07:38:50.650357 12754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 07:38:50.653863 12769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:50.655561 12754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:50.655812 12754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "d43313e77842451dba471ff89e02758d"
format_stamp: "Formatted at 2026-09-26 07:38:50 on dist-test-slave-qlq8"
server_key: "a8bd2dafea88172b5ad9768dd51f8a9b"
server_key_iv: "05fedc9aafab85d566846ed0545e53bb"
server_key_version: "encryptionkey@0"
I20260926 07:38:50.656105 12754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:50.681435 12754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:50.686079 12754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:50.686405 12754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:50.707038 12754 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:37499
I20260926 07:38:50.707093 12820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:37499 every 8 connection(s)
I20260926 07:38:50.708972 12754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:50.711480 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12754
I20260926 07:38:50.711696 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:50.712180 23413 external_mini_cluster.cc:1428] Setting key 82970785c0a23d0170f35ca7ff35a0b1
I20260926 07:38:50.713289 12821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:50.727900 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:50.714178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57931 (local address 127.22.221.126:37499)
0926 07:38:50.714954 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:50.714969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:50.715014 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:38:50.716812 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:50.716839 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:50.717375 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:50.717845 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:50.718756 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:50.720888 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:50.722275 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:50.723347 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:50.723527 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:50.724938 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:50.724950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:50.724971 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:50.725728 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:50.725831 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:50.725863 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:50.726030 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:50.726347 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:38:50.726541 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":329,"threads_started":1}
I20260926 07:38:50.728883 12821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d: Bootstrap starting.
I20260926 07:38:50.731907 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:50.732334 12821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:50.733558 12821 log.cc:824] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:50.737020 12821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d: No bootstrap required, opened a new log
I20260926 07:38:50.744045 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:50.737081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34053 (local address 127.22.221.126:37499)
0926 07:38:50.737245 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:50.737252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:50.737726 (+ 474us) server_negotiation.cc:407] Connection header received
0926 07:38:50.737876 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:50.737884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:50.737962 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:50.738071 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:50.738542 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:50.739778 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:50.741160 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:50.741829 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:50.741969 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:50.743066 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:50.743073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:50.743083 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:50.743109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:50.743162 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:50.743182 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:50.743340 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:50.743661 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 07:38:50.743715 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:38:50.746438 12821 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43313e77842451dba471ff89e02758d" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37499 } }
I20260926 07:38:50.746778 12821 raft_consensus.cc:374] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:50.746855 12821 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43313e77842451dba471ff89e02758d, State: Initialized, Role: FOLLOWER
I20260926 07:38:50.747432 12821 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43313e77842451dba471ff89e02758d" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37499 } }
I20260926 07:38:50.747634 12821 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:50.747727 12821 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:50.747862 12821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:50.749943 12821 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43313e77842451dba471ff89e02758d" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37499 } }
I20260926 07:38:50.750499 12821 leader_election.cc:302] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43313e77842451dba471ff89e02758d; no voters:
I20260926 07:38:50.751135 12821 leader_election.cc:288] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:50.751304 12831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:50.752040 12831 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [term 1 LEADER]: Becoming Leader. State: Replica: d43313e77842451dba471ff89e02758d, State: Running, Role: LEADER
I20260926 07:38:50.752617 12831 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43313e77842451dba471ff89e02758d" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37499 } }
I20260926 07:38:50.753446 12821 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:50.755635 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d43313e77842451dba471ff89e02758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43313e77842451dba471ff89e02758d" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37499 } } }
I20260926 07:38:50.756070 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:50.755996 12833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43313e77842451dba471ff89e02758d. Latest consensus state: current_term: 1 leader_uuid: "d43313e77842451dba471ff89e02758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43313e77842451dba471ff89e02758d" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 37499 } } }
I20260926 07:38:50.756274 12833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:50.757578 12837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:50.762790 12837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:50.771345 12837 catalog_manager.cc:1347] Generated new cluster ID: dd45f7a11544418f8f3cfa72bcf98b12
I20260926 07:38:50.771464 12837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:50.804448 12837 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:50.806327 12837 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:50.814604 12837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d43313e77842451dba471ff89e02758d: Generated new TSK 0
I20260926 07:38:50.815661 12837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:50.830641 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12754
W20260926 07:38:50.831822 12816 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57931
W20260926 07:38:50.832572 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12754: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:50.832721 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12754
W20260926 07:38:50.849601 12739 connection.cc:535] client connection to 127.22.221.126:37499 recv error: Network error: failed to read from TLS socket (remote: 127.22.221.126:37499): Connection reset by peer (error 104)
2026-09-26T07:38:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:50 dist-test-slave-qlq8 krb5kdc[12852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:50 dist-test-slave-qlq8 krb5kdc[12852](info): set up 2 sockets
Sep 26 07:38:50 dist-test-slave-qlq8 krb5kdc[12852](info): commencing operation
krb5kdc: starting...
W20260926 07:38:51.931854 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:52 dist-test-slave-qlq8 krb5kdc[12852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:52.193899 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46425
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:46425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:52.521450 12865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:52.521750 12865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:52.521914 12865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:52.529155 12865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:52.529269 12865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:52.529381 12865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:52.529488 12865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:52.529574 12865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:52.539472 12865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:46425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:52.540791 12865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:52.542093 12865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:52.553658 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:52.553730 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:52.553758 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:52.554617 12865 server_base.cc:1034] running on GCE node
I20260926 07:38:52.555437 12865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:52.556684 12865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:52.557947 12865 hybrid_clock.cc:648] HybridClock initialized: now 1790408332557901 us; error 32 us; skew 500 ppm
I20260926 07:38:52.558311 12865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:52 dist-test-slave-qlq8 krb5kdc[12852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408332, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:52.561352 12865 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:52.562887 12865 webserver.cc:460] Webserver started at http://127.22.221.126:46013/ using document root <none> and password file <none>
I20260926 07:38:52.563637 12865 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:52.563737 12865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:52.564064 12865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:52.566912 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "b077e4d482dc42458340121e6bdbec4c"
format_stamp: "Formatted at 2026-09-26 07:38:52 on dist-test-slave-qlq8"
server_key: "17cabd06be17dc1b5fd0c1461bcced93"
server_key_iv: "a38cdc381a81d2aa957e7160de05a921"
server_key_version: "encryptionkey@0"
I20260926 07:38:52.567713 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "b077e4d482dc42458340121e6bdbec4c"
format_stamp: "Formatted at 2026-09-26 07:38:52 on dist-test-slave-qlq8"
server_key: "17cabd06be17dc1b5fd0c1461bcced93"
server_key_iv: "a38cdc381a81d2aa957e7160de05a921"
server_key_version: "encryptionkey@0"
I20260926 07:38:52.572593 12865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:38:52.575839 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:52.577245 12865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:38:52.577414 12865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "b077e4d482dc42458340121e6bdbec4c"
format_stamp: "Formatted at 2026-09-26 07:38:52 on dist-test-slave-qlq8"
server_key: "17cabd06be17dc1b5fd0c1461bcced93"
server_key_iv: "a38cdc381a81d2aa957e7160de05a921"
server_key_version: "encryptionkey@0"
I20260926 07:38:52.577584 12865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:52.583221 12865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:52.587286 12865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:52.587581 12865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:52.606102 12865 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:46425
I20260926 07:38:52.606146 12931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:46425 every 8 connection(s)
I20260926 07:38:52.607493 12865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:52.608423 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12865
I20260926 07:38:52.608621 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:52.609143 23413 external_mini_cluster.cc:1428] Setting key 3de0972c943df63175faeb6c31e6c7b9
I20260926 07:38:52.611917 12932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:52.616556 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:52.610937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46969 (local address 127.22.221.126:46425)
0926 07:38:52.611725 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:52.611736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:52.611774 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:38:52.613377 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:52.613398 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:52.613714 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:52.614011 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:52.615042 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:52.615048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:52.615060 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:52.615079 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:52.615153 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:52.615172 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:52.615395 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:52.615502 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 07:38:52.615690 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":375,"threads_started":1}
I20260926 07:38:52.619997 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:52.624332 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c: Bootstrap starting.
I20260926 07:38:52.628075 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:52.625384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36607 (local address 127.22.221.126:46425)
0926 07:38:52.625573 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:52.625579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:52.626056 (+ 477us) server_negotiation.cc:407] Connection header received
0926 07:38:52.626195 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:52.626202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:52.626265 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:52.626390 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:52.627321 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:52.627326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:52.627330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:52.627347 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:52.627395 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:52.627416 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:52.627569 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:52.627688 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 07:38:52.627744 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:38:52.628098 12932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:52.629222 12932 log.cc:824] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:52.631932 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c: No bootstrap required, opened a new log
I20260926 07:38:52.639190 12932 raft_consensus.cc:348] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077e4d482dc42458340121e6bdbec4c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46425 } }
I20260926 07:38:52.639681 12932 raft_consensus.cc:374] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:52.639798 12932 raft_consensus.cc:729] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b077e4d482dc42458340121e6bdbec4c, State: Initialized, Role: FOLLOWER
I20260926 07:38:52.640388 12932 consensus_queue.cc:260] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077e4d482dc42458340121e6bdbec4c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46425 } }
I20260926 07:38:52.640650 12932 raft_consensus.cc:388] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:52.640805 12932 raft_consensus.cc:482] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:52.640992 12932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:52.643316 12932 raft_consensus.cc:504] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077e4d482dc42458340121e6bdbec4c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46425 } }
I20260926 07:38:52.643863 12932 leader_election.cc:302] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b077e4d482dc42458340121e6bdbec4c; no voters:
I20260926 07:38:52.644474 12932 leader_election.cc:288] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:52.644630 12942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:52.645334 12942 raft_consensus.cc:686] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [term 1 LEADER]: Becoming Leader. State: Replica: b077e4d482dc42458340121e6bdbec4c, State: Running, Role: LEADER
I20260926 07:38:52.645890 12942 consensus_queue.cc:237] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077e4d482dc42458340121e6bdbec4c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46425 } }
I20260926 07:38:52.646633 12932 sys_catalog.cc:564] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:52.648604 12943 sys_catalog.cc:455] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b077e4d482dc42458340121e6bdbec4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077e4d482dc42458340121e6bdbec4c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46425 } } }
I20260926 07:38:52.648950 12943 sys_catalog.cc:458] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:52.649837 12947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:52.649772 12944 sys_catalog.cc:455] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b077e4d482dc42458340121e6bdbec4c. Latest consensus state: current_term: 1 leader_uuid: "b077e4d482dc42458340121e6bdbec4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b077e4d482dc42458340121e6bdbec4c" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46425 } } }
I20260926 07:38:52.650082 12944 sys_catalog.cc:458] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:52.656281 12947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:52.665320 12947 catalog_manager.cc:1347] Generated new cluster ID: fd32fb4bd7074c208c5ccdc7f9b7f7af
I20260926 07:38:52.665431 12947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:52.701622 12947 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:52.702047 12947 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:52.709419 12947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b077e4d482dc42458340121e6bdbec4c: Generated new TSK 0
I20260926 07:38:52.710222 12947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:52.719899 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12865
W20260926 07:38:52.721035 12927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46969
W20260926 07:38:52.721764 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:52.721899 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12865
2026-09-26T07:38:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:52 dist-test-slave-qlq8 krb5kdc[12963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:52 dist-test-slave-qlq8 krb5kdc[12963](info): set up 2 sockets
Sep 26 07:38:52 dist-test-slave-qlq8 krb5kdc[12963](info): commencing operation
krb5kdc: starting...
W20260926 07:38:53.827857 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:54 dist-test-slave-qlq8 krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:54.099098 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:34677
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:54.428297 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:54.428632 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:54.428818 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:54.436174 12976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:54.436286 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:54.436371 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:54.436455 12976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:54.436534 12976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:54.446446 12976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:34677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:54.447749 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:54.449065 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:54.461215 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:54.461197 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:54.461241 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:54.462634 12976 server_base.cc:1034] running on GCE node
I20260926 07:38:54.463721 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:54.465999 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:54.467271 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790408334467143 us; error 94 us; skew 500 ppm
I20260926 07:38:54.467738 12976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:54 dist-test-slave-qlq8 krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408334, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:54.471825 12976 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:54.473400 12976 webserver.cc:460] Webserver started at http://127.22.221.126:34247/ using document root <none> and password file <none>
I20260926 07:38:54.474121 12976 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:54.474225 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:54.474469 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:54.477463 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "9b7650b3869b40c49c2602d33c994d8f"
format_stamp: "Formatted at 2026-09-26 07:38:54 on dist-test-slave-qlq8"
server_key: "24e975e29b60f1d7c3a3cc16af3ed71e"
server_key_iv: "f51543efb069b8b7980feca0c5eb755c"
server_key_version: "encryptionkey@0"
I20260926 07:38:54.478361 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "9b7650b3869b40c49c2602d33c994d8f"
format_stamp: "Formatted at 2026-09-26 07:38:54 on dist-test-slave-qlq8"
server_key: "24e975e29b60f1d7c3a3cc16af3ed71e"
server_key_iv: "f51543efb069b8b7980feca0c5eb755c"
server_key_version: "encryptionkey@0"
I20260926 07:38:54.483542 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:38:54.486847 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:54.488210 12976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:38:54.488350 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "9b7650b3869b40c49c2602d33c994d8f"
format_stamp: "Formatted at 2026-09-26 07:38:54 on dist-test-slave-qlq8"
server_key: "24e975e29b60f1d7c3a3cc16af3ed71e"
server_key_iv: "f51543efb069b8b7980feca0c5eb755c"
server_key_version: "encryptionkey@0"
I20260926 07:38:54.488512 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:54.541706 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:54.546203 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:54.546442 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:54.566020 12976 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:34677
I20260926 07:38:54.566021 13042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:34677 every 8 connection(s)
I20260926 07:38:54.567461 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:54.571139 13043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:54.575632 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 12976
I20260926 07:38:54.575812 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:54.576380 23413 external_mini_cluster.cc:1428] Setting key 0ec35fc8b14adbfde989e63c8514fd34
I20260926 07:38:54.582647 13043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f: Bootstrap starting.
I20260926 07:38:54.586184 13043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:54.587443 13043 log.cc:824] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:54.590968 13043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f: No bootstrap required, opened a new log
Sep 26 07:38:54 dist-test-slave-qlq8 krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:38:54.601676 13043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7650b3869b40c49c2602d33c994d8f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34677 } }
I20260926 07:38:54.602126 13043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:54.602238 13043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7650b3869b40c49c2602d33c994d8f, State: Initialized, Role: FOLLOWER
I20260926 07:38:54.602759 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:54.578178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40623 (local address 127.22.221.126:34677)
0926 07:38:54.579017 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:54.579038 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:54.579081 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:38:54.581122 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:54.581144 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:54.581550 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:54.581883 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:54.583788 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:54.585697 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:54.587645 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:54.588776 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:54.588958 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:54.593966 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:54.593977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:54.593998 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:54.594059 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:54.598197 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:54.598912 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:54.598920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:54.598929 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:54.599151 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:54.599612 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:54.599626 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:54.599631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:54.600351 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:54.600671 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:54.601009 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 07:38:54.601435 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":421,"threads_started":1}
I20260926 07:38:54.602874 13043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7650b3869b40c49c2602d33c994d8f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34677 } }
I20260926 07:38:54.603163 13043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:54.603390 13043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:54.603577 13043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:54.606233 13043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7650b3869b40c49c2602d33c994d8f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34677 } }
I20260926 07:38:54.606510 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:54.606807 13043 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7650b3869b40c49c2602d33c994d8f; no voters:
I20260926 07:38:54.607497 13043 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:54.607590 13048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:54.609961 13043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:54.611781 13048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [term 1 LEADER]: Becoming Leader. State: Replica: 9b7650b3869b40c49c2602d33c994d8f, State: Running, Role: LEADER
I20260926 07:38:54.612473 13048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7650b3869b40c49c2602d33c994d8f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34677 } }
I20260926 07:38:54.615857 13049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b7650b3869b40c49c2602d33c994d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7650b3869b40c49c2602d33c994d8f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34677 } } }
I20260926 07:38:54.615842 13050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7650b3869b40c49c2602d33c994d8f. Latest consensus state: current_term: 1 leader_uuid: "9b7650b3869b40c49c2602d33c994d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7650b3869b40c49c2602d33c994d8f" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 34677 } } }
I20260926 07:38:54.616328 13049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:54.616360 13050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:54.624037 13061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:54.631810 13061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:54.641938 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:54.612800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38825 (local address 127.22.221.126:34677)
0926 07:38:54.613004 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:54.613012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:54.613559 (+ 547us) server_negotiation.cc:407] Connection header received
0926 07:38:54.613757 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:54.613764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:54.613860 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:54.613972 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:54.623784 (+ 9812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:54.625270 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:54.627439 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:54.628148 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:54.628249 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:54.631681 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:54.631688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:54.631692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:38:54.631733 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:54.636670 (+ 4937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:54.637888 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:54.637897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:54.637903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:54.638099 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:38:54.638669 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:38:54.638677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:38:54.638683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:38:54.639283 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:54.639613 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:54.641100 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0926 07:38:54.641385 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:38:54.645843 13061 catalog_manager.cc:1347] Generated new cluster ID: bf9984a7763b4c81899db037c5f01da5
I20260926 07:38:54.645979 13061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:54.667145 13061 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:54.669018 13061 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:54.676978 13061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b7650b3869b40c49c2602d33c994d8f: Generated new TSK 0
I20260926 07:38:54.678009 13061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:54.684767 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 12976
I20260926 07:38:54.686010 13038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:38:54.778481 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 12976
2026-09-26T07:38:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:54 dist-test-slave-qlq8 krb5kdc[13076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:54 dist-test-slave-qlq8 krb5kdc[13076](info): set up 2 sockets
Sep 26 07:38:54 dist-test-slave-qlq8 krb5kdc[13076](info): commencing operation
krb5kdc: starting...
W20260926 07:38:55.879797 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:56 dist-test-slave-qlq8 krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:56.146791 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39249
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:37317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:39249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:56.487454 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:56.487761 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:56.487957 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:56.495709 13089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:56.495826 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:56.495903 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:56.496012 13089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:56.496078 13089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:56.507715 13089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:37317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:39249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:56.509085 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:56.510514 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:56.523193 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:56.523303 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:56.523307 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:56.524371 13089 server_base.cc:1034] running on GCE node
I20260926 07:38:56.525238 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:56.526666 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:56.527863 13089 hybrid_clock.cc:648] HybridClock initialized: now 1790408336527828 us; error 36 us; skew 500 ppm
I20260926 07:38:56.528170 13089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:56 dist-test-slave-qlq8 krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408336, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:56.536725 13089 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:56.538318 13089 webserver.cc:460] Webserver started at http://127.22.221.126:36131/ using document root <none> and password file <none>
I20260926 07:38:56.539042 13089 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:56.539181 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:56.539558 13089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:56.542563 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "ed3f24d5af6a49379ae49c121ce8f3ed"
format_stamp: "Formatted at 2026-09-26 07:38:56 on dist-test-slave-qlq8"
server_key: "63c646abcd3776056a8f4dbe8cadf233"
server_key_iv: "f543ff3098cca956c4691755231bce2c"
server_key_version: "encryptionkey@0"
I20260926 07:38:56.543460 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "ed3f24d5af6a49379ae49c121ce8f3ed"
format_stamp: "Formatted at 2026-09-26 07:38:56 on dist-test-slave-qlq8"
server_key: "63c646abcd3776056a8f4dbe8cadf233"
server_key_iv: "f543ff3098cca956c4691755231bce2c"
server_key_version: "encryptionkey@0"
I20260926 07:38:56.548799 13089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 07:38:56.552340 13104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:56.553721 13089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:38:56.553920 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "ed3f24d5af6a49379ae49c121ce8f3ed"
format_stamp: "Formatted at 2026-09-26 07:38:56 on dist-test-slave-qlq8"
server_key: "63c646abcd3776056a8f4dbe8cadf233"
server_key_iv: "f543ff3098cca956c4691755231bce2c"
server_key_version: "encryptionkey@0"
I20260926 07:38:56.554132 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:56.569135 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:56.573748 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:56.574060 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:56.593492 13089 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:39249
I20260926 07:38:56.593585 13155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:39249 every 8 connection(s)
I20260926 07:38:56.594907 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:56.598645 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:56.602603 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13089
I20260926 07:38:56.602807 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:56.603442 23413 external_mini_cluster.cc:1428] Setting key 49ec6c81e71d5c2f40a56794a687d819
I20260926 07:38:56.610904 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed: Bootstrap starting.
I20260926 07:38:56.614131 13156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:56.615278 13156 log.cc:824] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:56.616261 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:56.605173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46849 (local address 127.22.221.126:39249)
0926 07:38:56.605921 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:56.605933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:56.605975 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:38:56.607647 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:56.607669 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:56.608076 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:56.608479 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:56.609048 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:56.610346 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:56.611831 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:56.612528 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:56.612737 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:56.613911 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:56.613919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:56.613931 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:56.614522 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:56.614597 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:56.614621 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:56.614747 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:56.614975 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 07:38:56.615269 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
I20260926 07:38:56.618033 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed: No bootstrap required, opened a new log
I20260926 07:38:56.619807 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:56.626305 13156 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39249 } }
I20260926 07:38:56.626761 13156 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:56.626909 13156 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3f24d5af6a49379ae49c121ce8f3ed, State: Initialized, Role: FOLLOWER
I20260926 07:38:56.627521 13156 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39249 } }
I20260926 07:38:56.627759 13156 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:56.627913 13156 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:56.628082 13156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:56.630892 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:56.624117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35221 (local address 127.22.221.126:39249)
0926 07:38:56.624306 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:56.624312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:56.624792 (+ 480us) server_negotiation.cc:407] Connection header received
0926 07:38:56.624944 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:56.624951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:56.625030 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:56.625140 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:56.625606 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:56.626661 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:56.628393 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:38:56.629075 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:38:56.629219 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:38:56.630207 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:56.630214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:56.630220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:56.630240 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:56.630283 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:56.630301 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:56.630404 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:56.630494 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:38:56.630579 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:38:56.631181 13156 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39249 } }
I20260926 07:38:56.631829 13156 leader_election.cc:302] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed3f24d5af6a49379ae49c121ce8f3ed; no voters:
I20260926 07:38:56.632427 13156 leader_election.cc:288] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:56.632553 13166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:56.633309 13166 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [term 1 LEADER]: Becoming Leader. State: Replica: ed3f24d5af6a49379ae49c121ce8f3ed, State: Running, Role: LEADER
I20260926 07:38:56.633926 13166 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39249 } }
I20260926 07:38:56.634691 13156 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:56.636757 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39249 } } }
I20260926 07:38:56.637147 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:56.639425 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed3f24d5af6a49379ae49c121ce8f3ed. Latest consensus state: current_term: 1 leader_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3f24d5af6a49379ae49c121ce8f3ed" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39249 } } }
I20260926 07:38:56.639791 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:56.646272 13173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:56.650837 13173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:56.659540 13173 catalog_manager.cc:1347] Generated new cluster ID: 8721298552b84ef7b7b340b968225e7e
I20260926 07:38:56.659672 13173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:56.668505 13173 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:56.669783 13173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:56.686159 13173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed3f24d5af6a49379ae49c121ce8f3ed: Generated new TSK 0
I20260926 07:38:56.687067 13173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:56.698413 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13089
W20260926 07:38:56.699678 13151 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46849
W20260926 07:38:56.700335 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13089: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:56.700489 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13089
2026-09-26T07:38:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:56 dist-test-slave-qlq8 krb5kdc[13187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:56 dist-test-slave-qlq8 krb5kdc[13187](info): set up 2 sockets
Sep 26 07:38:56 dist-test-slave-qlq8 krb5kdc[13187](info): commencing operation
krb5kdc: starting...
W20260926 07:38:57.823863 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:57 dist-test-slave-qlq8 krb5kdc[13187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:38:58.094658 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39689
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:39689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:38:58.427551 13200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:38:58.427852 13200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:38:58.428051 13200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:38:58.435465 13200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:38:58.435573 13200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:38:58.435711 13200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:38:58.435818 13200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:38:58.435882 13200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:38:58.445691 13200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:39689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:38:58.446964 13200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:38:58.448266 13200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:38:58.460119 13208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:58.460180 13205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:38:58.460180 13206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:38:58.461035 13200 server_base.cc:1034] running on GCE node
I20260926 07:38:58.461889 13200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:38:58.463470 13200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:38:58.464733 13200 hybrid_clock.cc:648] HybridClock initialized: now 1790408338464656 us; error 61 us; skew 500 ppm
I20260926 07:38:58.465201 13200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:38:58 dist-test-slave-qlq8 krb5kdc[13187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408338, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:38:58.469655 13200 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:38:58.471923 13200 webserver.cc:460] Webserver started at http://127.22.221.126:39935/ using document root <none> and password file <none>
I20260926 07:38:58.472743 13200 fs_manager.cc:359] Metadata directory not provided
I20260926 07:38:58.472883 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:38:58.473284 13200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:38:58.476366 13200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "41fb81a6b9014d7894c0b2aec8a44629"
format_stamp: "Formatted at 2026-09-26 07:38:58 on dist-test-slave-qlq8"
server_key: "7402a1d6d1bb2ce84b329220d257757b"
server_key_iv: "7be0395cdc7045e654586a1269e61e6d"
server_key_version: "encryptionkey@0"
I20260926 07:38:58.477138 13200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "41fb81a6b9014d7894c0b2aec8a44629"
format_stamp: "Formatted at 2026-09-26 07:38:58 on dist-test-slave-qlq8"
server_key: "7402a1d6d1bb2ce84b329220d257757b"
server_key_iv: "7be0395cdc7045e654586a1269e61e6d"
server_key_version: "encryptionkey@0"
I20260926 07:38:58.482254 13200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 07:38:58.485730 13215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:38:58.487028 13200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:38:58.487210 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "41fb81a6b9014d7894c0b2aec8a44629"
format_stamp: "Formatted at 2026-09-26 07:38:58 on dist-test-slave-qlq8"
server_key: "7402a1d6d1bb2ce84b329220d257757b"
server_key_iv: "7be0395cdc7045e654586a1269e61e6d"
server_key_version: "encryptionkey@0"
I20260926 07:38:58.487468 13200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:38:58.493170 13200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:38:58.497047 13200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:38:58.497327 13200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:38:58.517863 13200 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:39689
I20260926 07:38:58.517891 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:39689 every 8 connection(s)
I20260926 07:38:58.519347 13200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:38:58.520918 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13200
I20260926 07:38:58.521126 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:38:58.521567 23413 external_mini_cluster.cc:1428] Setting key 5e288bfcfb9106c26118b80af87d5f51
I20260926 07:38:58.523635 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:38:58.530467 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:58.523393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42933 (local address 127.22.221.126:39689)
0926 07:38:58.524262 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:58.524277 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:58.524325 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:38:58.526510 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:58.526537 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:58.526994 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:58.527461 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:58.528457 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:58.528465 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:58.528484 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:58.528500 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:58.528592 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:58.528620 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:58.528829 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:58.528975 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 07:38:58.529171 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":372,"threads_started":1}
I20260926 07:38:58.534703 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:38:58.537192 13267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629: Bootstrap starting.
I20260926 07:38:58.543735 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:38:58.540550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44725 (local address 127.22.221.126:39689)
0926 07:38:58.540735 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:38:58.540741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:38:58.541487 (+ 746us) server_negotiation.cc:407] Connection header received
0926 07:38:58.541559 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:38:58.541564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:38:58.541654 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:38:58.541777 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:38:58.542977 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:38:58.542985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:38:58.542991 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:38:58.543012 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:38:58.543079 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:38:58.543102 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:38:58.543214 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:38:58.543351 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:38:58.543413 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:38:58.544041 13267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629: Neither blocks nor log segments found. Creating new log.
I20260926 07:38:58.545462 13267 log.cc:824] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629: Log is configured to *not* fsync() on all Append() calls
I20260926 07:38:58.549216 13267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629: No bootstrap required, opened a new log
I20260926 07:38:58.561343 13267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fb81a6b9014d7894c0b2aec8a44629" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39689 } }
I20260926 07:38:58.561896 13267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:38:58.562011 13267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fb81a6b9014d7894c0b2aec8a44629, State: Initialized, Role: FOLLOWER
I20260926 07:38:58.562799 13267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fb81a6b9014d7894c0b2aec8a44629" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39689 } }
I20260926 07:38:58.563040 13267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:38:58.563169 13267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:38:58.563380 13267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:38:58.566776 13267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fb81a6b9014d7894c0b2aec8a44629" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39689 } }
I20260926 07:38:58.567406 13267 leader_election.cc:302] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41fb81a6b9014d7894c0b2aec8a44629; no voters:
I20260926 07:38:58.568120 13267 leader_election.cc:288] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:38:58.568243 13277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:38:58.568910 13277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [term 1 LEADER]: Becoming Leader. State: Replica: 41fb81a6b9014d7894c0b2aec8a44629, State: Running, Role: LEADER
I20260926 07:38:58.569504 13277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fb81a6b9014d7894c0b2aec8a44629" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39689 } }
I20260926 07:38:58.570703 13267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:38:58.572664 13279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41fb81a6b9014d7894c0b2aec8a44629. Latest consensus state: current_term: 1 leader_uuid: "41fb81a6b9014d7894c0b2aec8a44629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fb81a6b9014d7894c0b2aec8a44629" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39689 } } }
I20260926 07:38:58.573192 13279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:58.572764 13278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41fb81a6b9014d7894c0b2aec8a44629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fb81a6b9014d7894c0b2aec8a44629" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 39689 } } }
I20260926 07:38:58.573891 13278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629 [sys.catalog]: This master's current role is: LEADER
I20260926 07:38:58.576033 13283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:38:58.581485 13283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:38:58.591792 13283 catalog_manager.cc:1347] Generated new cluster ID: 416a64b801ff45a19e6194bdb85cefea
I20260926 07:38:58.591934 13283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:38:58.608644 13283 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:38:58.609091 13283 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:38:58.616981 13283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41fb81a6b9014d7894c0b2aec8a44629: Generated new TSK 0
I20260926 07:38:58.618007 13283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:38:58.631108 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13200
W20260926 07:38:58.632993 13262 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42933
W20260926 07:38:58.633724 23413 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:38:58.633877 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13200
2026-09-26T07:38:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1936 ms)
[----------] 18 tests from AuthTokenIssuingTest (35350 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:38:58 dist-test-slave-qlq8 krb5kdc[13298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:38:58 dist-test-slave-qlq8 krb5kdc[13298](info): set up 2 sockets
Sep 26 07:38:58 dist-test-slave-qlq8 krb5kdc[13298](info): commencing operation
krb5kdc: starting...
W20260926 07:38:59.747957 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:38:59 dist-test-slave-qlq8 krb5kdc[13298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:39:00.028083 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46037
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:46037,127.22.221.125:42455,127.22.221.124:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:39:00.354425 13311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:39:00.354790 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:39:00.354919 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:39:00.355050 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:39:00.362387 13311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:39:00.362486 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:39:00.362592 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:39:00.362669 13311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:39:00.362756 13311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:39:00.372994 13311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:46037,127.22.221.125:42455,127.22.221.124:39709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:39:00.374363 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:39:00.375767 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:39:00.388324 13319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:00.388409 13317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:00.388356 13316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:39:00.389199 13311 server_base.cc:1034] running on GCE node
I20260926 07:39:00.389930 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:39:00.391434 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:39:00.392760 13311 hybrid_clock.cc:648] HybridClock initialized: now 1790408340392673 us; error 98 us; skew 500 ppm
I20260926 07:39:00.393180 13311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:39:00 dist-test-slave-qlq8 krb5kdc[13298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:39:00.396651 13311 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:39:00.398300 13311 webserver.cc:460] Webserver started at http://127.22.221.126:39483/ using document root <none> and password file <none>
I20260926 07:39:00.399019 13311 fs_manager.cc:359] Metadata directory not provided
I20260926 07:39:00.399118 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:39:00.399489 13311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:39:00.402658 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "e445b57f42e54fc9bedc6237bbe24ad5"
format_stamp: "Formatted at 2026-09-26 07:39:00 on dist-test-slave-qlq8"
server_key: "e96e9dc8aa2e5cfb695cff4cd40a7f0c"
server_key_iv: "efbd169f0dc8512e9b72d055fa9df16c"
server_key_version: "encryptionkey@0"
I20260926 07:39:00.403627 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "e445b57f42e54fc9bedc6237bbe24ad5"
format_stamp: "Formatted at 2026-09-26 07:39:00 on dist-test-slave-qlq8"
server_key: "e96e9dc8aa2e5cfb695cff4cd40a7f0c"
server_key_iv: "efbd169f0dc8512e9b72d055fa9df16c"
server_key_version: "encryptionkey@0"
I20260926 07:39:00.408916 13311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:39:00.412384 13326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:39:00.413944 13311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:39:00.414136 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "e445b57f42e54fc9bedc6237bbe24ad5"
format_stamp: "Formatted at 2026-09-26 07:39:00 on dist-test-slave-qlq8"
server_key: "e96e9dc8aa2e5cfb695cff4cd40a7f0c"
server_key_iv: "efbd169f0dc8512e9b72d055fa9df16c"
server_key_version: "encryptionkey@0"
I20260926 07:39:00.414361 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:39:00.436830 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:39:00.441026 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:39:00.441339 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:39:00.460636 13311 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:46037
I20260926 07:39:00.460644 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:46037 every 8 connection(s)
I20260926 07:39:00.462054 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:39:00.464452 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13311
I20260926 07:39:00.464666 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:39:00.465415 23413 external_mini_cluster.cc:1428] Setting key c344b7e2800476d14376d566fe205526
I20260926 07:39:00.474802 13378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:39:00.485304 13378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.125@KRBTEST.COM" created.
Sep 26 07:39:00 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:39:00.509999 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.489911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.22.221.126:46037)
0926 07:39:00.490879 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:00.490899 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:00.491382 (+ 483us) server_negotiation.cc:407] Connection header received
0926 07:39:00.493487 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:00.493498 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:00.493716 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:00.493935 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:00.495218 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.496604 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.498427 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.498997 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.499203 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:00.502490 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:00.502498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:00.502510 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:00.502542 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:00.505400 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:00.506099 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:00.506114 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:00.506116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:00.506260 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:00.506631 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:00.506639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:00.506643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:00.507384 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:00.507604 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:00.508152 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 07:39:00.508509 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":696,"thread_start_us":552,"threads_started":1}
I20260926 07:39:00.510030 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.489047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:46037 (local address 127.0.0.1:37330)
0926 07:39:00.491172 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.491207 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:00.493094 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:00.493950 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:00.493957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:00.494680 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:00.495008 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:00.495026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.496771 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.496776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:00.498275 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:00.498287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.499249 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.499257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:00.499437 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:00.500113 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:00.500140 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:00.502274 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:00.505551 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:00.505561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:00.505573 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:00.505969 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:00.506367 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:00.506374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:00.506377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:00.506528 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:00.507549 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:00.507555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:00.507762 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 07:39:00.508021 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 07:39:00.508624 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1908,"mutex_wait_us":211,"thread_start_us":1681,"threads_started":1}
I20260926 07:39:00.513448 13378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:00.514551 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.514142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37862)
0926 07:39:00.514316 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.514428 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 07:39:00.514842 13329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
I20260926 07:39:00.516685 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.516140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37872)
0926 07:39:00.516385 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.516488 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260926 07:39:00.516916 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:42455: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab.
Entry for principal kudu/127.22.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab.
I20260926 07:39:00.534896 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:42455
--webserver_interface=127.22.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab
--principal=kudu/127.22.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:46037,127.22.221.125:42455,127.22.221.124:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:39:00.566299 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } attempt: 1
I20260926 07:39:00.567586 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.567008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37888)
0926 07:39:00.567261 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.567417 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 07:39:00.568591 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.568192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37892)
0926 07:39:00.568362 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.568442 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 07:39:00.568796 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:42455: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
I20260926 07:39:00.637110 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } attempt: 2
I20260926 07:39:00.638387 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.637794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37894)
0926 07:39:00.638094 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.638212 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:39:00.639598 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.639096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37906)
0926 07:39:00.639335 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.639426 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:39:00.639822 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:42455: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
I20260926 07:39:00.731101 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } attempt: 3
I20260926 07:39:00.732319 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.731814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37916)
0926 07:39:00.732052 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.732160 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:39:00.733394 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.732988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37932)
0926 07:39:00.733146 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.733239 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 07:39:00.733608 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:42455: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
W20260926 07:39:00.866071 13385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:39:00.866438 13385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:39:00.866560 13385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:39:00.866693 13385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:39:00.874161 13385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:39:00.874260 13385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:39:00.874360 13385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:39:00.874459 13385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:39:00.874573 13385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:39:00.876861 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } attempt: 4
I20260926 07:39:00.878135 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.877517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37934)
0926 07:39:00.877841 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.877960 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 07:39:00.879307 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.878862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37944)
0926 07:39:00.879027 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.879122 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 07:39:00.879492 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:42455: Network error: Client connection negotiation failed: client connection to 127.22.221.125:42455: connect: Connection refused (error 111)
I20260926 07:39:00.885766 13385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:46037,127.22.221.125:42455,127.22.221.124:39709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab
--principal=kudu/127.22.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:39:00.887105 13385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:39:00.888499 13385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:39:00.900581 13390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:00.900686 13393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:00.900528 13391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:39:00.901465 13385 server_base.cc:1034] running on GCE node
I20260926 07:39:00.902271 13385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:39:00.903745 13385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:39:00.904990 13385 hybrid_clock.cc:648] HybridClock initialized: now 1790408340904923 us; error 46 us; skew 500 ppm
I20260926 07:39:00.905328 13385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:39:00 dist-test-slave-qlq8 krb5kdc[13298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:39:00.908550 13385 init.cc:377] Logged in from keytab as kudu/127.22.221.125@KRBTEST.COM (short username kudu)
I20260926 07:39:00.909983 13385 webserver.cc:460] Webserver started at http://127.22.221.125:36317/ using document root <none> and password file <none>
I20260926 07:39:00.910652 13385 fs_manager.cc:359] Metadata directory not provided
I20260926 07:39:00.910737 13385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:39:00.910982 13385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:39:00.914011 13385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data/instance:
uuid: "f2e35e0aacc041e0b3e20aa1e8aee626"
format_stamp: "Formatted at 2026-09-26 07:39:00 on dist-test-slave-qlq8"
server_key: "5ef1dcaaf516e47d26e85218d16d328e"
server_key_iv: "9b3ad00e66c661be149be7964b42f04c"
server_key_version: "encryptionkey@0"
I20260926 07:39:00.914754 13385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal/instance:
uuid: "f2e35e0aacc041e0b3e20aa1e8aee626"
format_stamp: "Formatted at 2026-09-26 07:39:00 on dist-test-slave-qlq8"
server_key: "5ef1dcaaf516e47d26e85218d16d328e"
server_key_iv: "9b3ad00e66c661be149be7964b42f04c"
server_key_version: "encryptionkey@0"
I20260926 07:39:00.920147 13385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 07:39:00.923522 13400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:39:00.924850 13385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:39:00.925037 13385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal
uuid: "f2e35e0aacc041e0b3e20aa1e8aee626"
format_stamp: "Formatted at 2026-09-26 07:39:00 on dist-test-slave-qlq8"
server_key: "5ef1dcaaf516e47d26e85218d16d328e"
server_key_iv: "9b3ad00e66c661be149be7964b42f04c"
server_key_version: "encryptionkey@0"
I20260926 07:39:00.925266 13385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:39:00.938637 13385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:39:00.942943 13385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:39:00.943279 13385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:39:00.962082 13385 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.125:42455
I20260926 07:39:00.962105 13451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.125:42455 every 8 connection(s)
I20260926 07:39:00.963514 13385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
I20260926 07:39:00.967243 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:39:00.970449 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13385
I20260926 07:39:00.970647 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-1/wal/instance
I20260926 07:39:00.971191 23413 external_mini_cluster.cc:1428] Setting key 74dbf680df3cce570cc27832fb4718a4
I20260926 07:39:00.975931 13452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:00 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:39:00.998685 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.979343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.22.221.126:46037)
0926 07:39:00.979594 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:00.979603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:00.980800 (+ 1197us) server_negotiation.cc:407] Connection header received
0926 07:39:00.982654 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:00.982660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:00.982754 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:00.982872 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:00.984867 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.985553 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.987926 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.988349 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.988454 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:00.992700 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:00.992705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:00.992709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:00.992746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:00.995543 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:00.996324 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:00.996334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:00.996337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:00.996455 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:00.996903 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:00.996908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:00.996911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:00.997207 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:00.997370 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:00.998117 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 07:39:00.998353 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for kudu/127.22.221.124@KRBTEST.COM; defaulting to no policy
I20260926 07:39:00.999850 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:00.979791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:46037 (local address 127.0.0.1:37338)
0926 07:39:00.980615 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:00.980669 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:00.982385 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:00.983118 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:00.983131 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:00.983900 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:00.984669 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:00.984696 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.985727 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.985732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:00.987760 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:00.987777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:00.988579 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:00.988588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:00.988903 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:00.989645 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:00.989678 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:00.992441 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:00.995693 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:00.995702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:00.995719 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:00.996185 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:00.996572 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:00.996577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:00.996580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:00.996778 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:00.997348 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:00.997356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:00.997723 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 07:39:00.998001 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 07:39:00.998561 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.124@KRBTEST.COM" created.
I20260926 07:39:01.001724 13452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:39:01.016284 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.002608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37950)
0926 07:39:01.002780 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.002804 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.002912 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.003900 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.003904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.004358 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.004485 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.004493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.005858 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.005866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.006714 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.006725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.007644 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.007650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.007844 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.008459 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.008476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.010482 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.013797 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.013802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.013805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.014173 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.014544 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.014548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.014552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.014684 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.015291 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.015300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.015446 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.015588 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.015872 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:39:01.016396 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.002828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.22.221.125:42455)
0926 07:39:01.003487 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.003492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.003515 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:39:01.003571 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.003577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.003752 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.003918 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.004666 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.005725 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.006850 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.007497 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.007639 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.010695 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.010706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.010720 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.010746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.013661 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.014302 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.014317 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.014320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.014429 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.014786 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.014794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.014797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.015145 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.015368 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.015722 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.015983 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":517,"thread_start_us":340,"threads_started":1}
I20260926 07:39:01.019179 13452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:01.020334 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.019879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36932)
0926 07:39:01.020055 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.020160 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 07:39:01.020642 13403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.022166 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.021838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36948)
0926 07:39:01.021986 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.022068 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:39:01.022401 13452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab.
Entry for principal kudu/127.22.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab.
I20260926 07:39:01.034363 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:39709
--webserver_interface=127.22.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab
--principal=kudu/127.22.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:46037,127.22.221.125:42455,127.22.221.124:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:39:01.071710 13452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 1
I20260926 07:39:01.073047 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.072463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36952)
0926 07:39:01.072722 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.072853 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:39:01.074203 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.073778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36954)
0926 07:39:01.073967 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.074073 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:39:01.074429 13452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.142690 13452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 2
I20260926 07:39:01.144177 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.143528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36966)
0926 07:39:01.143797 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.143945 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 07:39:01.145392 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.144961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36970)
0926 07:39:01.145153 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.145243 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 07:39:01.145633 13452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.178771 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } attempt: 5
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:39:01.194204 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.179610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37956)
0926 07:39:01.179892 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.179913 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.180052 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.180451 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.180459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.180883 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.181076 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.181086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.182499 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.182504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.183543 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.183557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.184596 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.184601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.184764 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.185566 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.185583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.187806 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.191647 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.191651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.191654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.192026 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.192495 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.192499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.192502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.192639 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.193360 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.193364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.193507 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.193679 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.193955 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260926 07:39:01.194391 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.179835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.22.221.125:42455)
0926 07:39:01.180045 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.180051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.180075 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:39:01.180161 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.180167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.180248 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.180364 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.181249 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.182342 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.183695 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.184423 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.184556 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.187998 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.188005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.188010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.188049 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.191511 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.192152 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.192161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.192167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.192343 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.192767 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.192774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.192781 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.193218 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.193400 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.193837 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.194089 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:39:01.195442 13378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:01.196403 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.196056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36972)
0926 07:39:01.196210 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.196282 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 07:39:01.197517 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.197054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36978)
0926 07:39:01.197249 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.197348 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:39:01.197764 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.236913 13452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 3
I20260926 07:39:01.238279 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.237690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:36994)
0926 07:39:01.237933 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.238080 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:39:01.239545 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.239015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37000)
0926 07:39:01.239291 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.239397 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260926 07:39:01.239814 13452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.249010 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 1
I20260926 07:39:01.250355 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.249682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37012)
0926 07:39:01.249950 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.250097 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:39:01.251427 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.251031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37014)
0926 07:39:01.251187 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.251300 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:39:01.251703 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.319914 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 2
I20260926 07:39:01.321108 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.320512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37028)
0926 07:39:01.320777 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.320912 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 07:39:01.322160 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.321761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37030)
0926 07:39:01.321927 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.322010 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:39:01.322412 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
W20260926 07:39:01.373402 13459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:39:01.373778 13459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:39:01.373886 13459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:39:01.374018 13459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:39:01.381553 13459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:39:01.381651 13459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:39:01.381737 13459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:39:01.381836 13459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:39:01.381922 13459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:39:01.383045 13452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 4
I20260926 07:39:01.384286 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.383776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37042)
0926 07:39:01.384024 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.384131 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:39:01.385428 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.385039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37052)
0926 07:39:01.385243 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.385315 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 07:39:01.385658 13452 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.392874 13459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:46037,127.22.221.125:42455,127.22.221.124:39709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab
--principal=kudu/127.22.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:39:01.394164 13459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:39:01.395617 13459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:39:01.407678 13464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:01.407646 13467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:01.407714 13465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:39:01.408671 13459 server_base.cc:1034] running on GCE node
I20260926 07:39:01.409564 13459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:39:01.410988 13459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:39:01.412210 13459 hybrid_clock.cc:648] HybridClock initialized: now 1790408341412145 us; error 40 us; skew 500 ppm
I20260926 07:39:01.412575 13459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:39:01.415917 13459 init.cc:377] Logged in from keytab as kudu/127.22.221.124@KRBTEST.COM (short username kudu)
I20260926 07:39:01.417482 13459 webserver.cc:460] Webserver started at http://127.22.221.124:43869/ using document root <none> and password file <none>
I20260926 07:39:01.418220 13459 fs_manager.cc:359] Metadata directory not provided
I20260926 07:39:01.418309 13459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:39:01.418610 13459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:39:01.421835 13459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data/instance:
uuid: "dcce89b49e414fcc8c711210cceed0dd"
format_stamp: "Formatted at 2026-09-26 07:39:01 on dist-test-slave-qlq8"
server_key: "a3ac6b194de1948e06813b380901ae2b"
server_key_iv: "e5984ee67fa52cf14a0d3034f181446f"
server_key_version: "encryptionkey@0"
I20260926 07:39:01.422600 13459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal/instance:
uuid: "dcce89b49e414fcc8c711210cceed0dd"
format_stamp: "Formatted at 2026-09-26 07:39:01 on dist-test-slave-qlq8"
server_key: "a3ac6b194de1948e06813b380901ae2b"
server_key_iv: "e5984ee67fa52cf14a0d3034f181446f"
server_key_version: "encryptionkey@0"
I20260926 07:39:01.427963 13459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 07:39:01.428683 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 3
I20260926 07:39:01.430019 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.429446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37066)
0926 07:39:01.429695 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.429802 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 07:39:01.431113 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.430744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37082)
0926 07:39:01.430892 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.430963 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:39:01.431366 13378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:39709: Network error: Client connection negotiation failed: client connection to 127.22.221.124:39709: connect: Connection refused (error 111)
I20260926 07:39:01.432111 13474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:39:01.433948 13459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:39:01.434147 13459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal
uuid: "dcce89b49e414fcc8c711210cceed0dd"
format_stamp: "Formatted at 2026-09-26 07:39:01 on dist-test-slave-qlq8"
server_key: "a3ac6b194de1948e06813b380901ae2b"
server_key_iv: "e5984ee67fa52cf14a0d3034f181446f"
server_key_version: "encryptionkey@0"
I20260926 07:39:01.434432 13459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:39:01.466650 13459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:39:01.471451 13459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:39:01.471748 13459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:39:01.492050 13459 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.124:39709
I20260926 07:39:01.492089 13525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.124:39709 every 8 connection(s)
I20260926 07:39:01.493548 13459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
I20260926 07:39:01.497716 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:39:01.500860 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13459
I20260926 07:39:01.501084 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407989013554-23413-0/minicluster-data/master-2/wal/instance
I20260926 07:39:01.501606 23413 external_mini_cluster.cc:1428] Setting key 8986413367cbbea42cab1112232b8401
I20260926 07:39:01.505709 13526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:39:01.528119 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.509537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.22.221.126:46037)
0926 07:39:01.511173 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.511179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.511196 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:39:01.511276 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.511281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.511351 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.511476 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.512903 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.514047 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.516054 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.516505 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.516608 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.520930 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.520936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.520939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.520978 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.524546 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.525472 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.525477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.525491 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.525650 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.526079 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.526084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.526087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.526447 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.526625 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.527380 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.527640 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":1413,"threads_started":1}
I20260926 07:39:01.529273 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.503837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.22.221.124:39709)
0926 07:39:01.504632 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.504647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.504696 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:39:01.506675 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.506709 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.507299 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.507776 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.509292 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.511491 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.513248 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.514339 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.514475 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.519722 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.519730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.519768 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.519813 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.523371 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.524406 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.524444 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.524454 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.524711 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.525178 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.525186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.525192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.526845 (+ 1653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.527173 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.527532 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.528243 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":424,"server-negotiator.queue_time_us":565,"thread_start_us":335,"threads_started":1}
I20260926 07:39:01.529296 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.509624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:46037 (local address 127.0.0.1:37350)
0926 07:39:01.510494 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.510546 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.510741 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.511543 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.511548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.512429 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.512738 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.512750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.514223 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.514227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.515873 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.515890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.516757 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.516766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.516999 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.517839 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.517863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.520686 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.524694 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.524704 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.524732 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.525317 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.525776 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.525779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.525782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.525931 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.526667 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.526678 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.526955 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.527247 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.527946 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":1,"thread_start_us":402,"threads_started":1}
I20260926 07:39:01.530972 13526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:39:01.554167 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.531856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.22.221.125:42455)
0926 07:39:01.532374 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.532381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.532400 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:39:01.532463 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.532468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.532573 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.532693 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.533608 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.534727 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.536567 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.537217 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.537348 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.547743 (+ 10395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.547752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.547760 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.547817 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.550789 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.551824 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.551829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.551831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.551954 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.552423 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.552428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.552431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.552828 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.553007 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.553620 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.553867 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260926 07:39:01.554188 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.531938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:42455 (local address 127.0.0.1:37962)
0926 07:39:01.532096 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.532118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.532225 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.532734 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.532743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.533228 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.533433 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.533446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.534876 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.534881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.536406 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.536419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.537370 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.537375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.537540 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.544590 (+ 7050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.544618 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.547452 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.550959 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.550967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.550972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.551698 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.552084 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.552089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.552093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.552275 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.552984 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.552988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.553240 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.553494 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.553782 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:39:01.555512 13526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:01.555838 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.532547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.22.221.126:46037)
0926 07:39:01.532719 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.532725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.532987 (+ 262us) server_negotiation.cc:407] Connection header received
0926 07:39:01.533089 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.533094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.533189 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.533300 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.534851 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.536072 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.537728 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.538458 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.538567 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.548869 (+ 10302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.548875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.548878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.548916 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.552290 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.553164 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.553171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.553175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.553343 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.553885 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.553916 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.553920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.554507 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.554835 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.555164 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.555455 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408341, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:39:01.569427 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.556249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37098)
0926 07:39:01.556413 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.556431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.556527 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.557026 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.557030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.557435 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.557637 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.557653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.558624 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.558632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.559568 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.559580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.560222 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.560227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.560458 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.561065 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.561082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.563316 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.566727 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.566734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.566738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.567328 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.567701 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.567705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.567707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.567829 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.568499 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.568502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.568632 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.568737 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.569084 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:39:01.569691 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.556489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.22.221.124:39709)
0926 07:39:01.556681 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.556687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.556709 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:39:01.556765 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.556771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.556850 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.556964 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.557801 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.558501 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.559699 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.560083 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.560222 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.563511 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.563530 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.563535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.563576 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.566589 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.567439 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.567444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.567447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.567589 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.567963 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.567971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.567977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.568365 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.568523 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.568882 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.569270 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":64}
I20260926 07:39:01.574064 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.557787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.22.221.125:42455)
0926 07:39:01.557971 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.557977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.558011 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:39:01.558061 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.558066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.558130 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.558231 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.559492 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.560202 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.561632 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.562117 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.562248 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.567223 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.567275 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.567280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.567324 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.571228 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.571906 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.571913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.571916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.572103 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.572617 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.572622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.572626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.572958 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.573249 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.573498 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.573733 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:39:01.575856 13526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: Bootstrap starting.
I20260926 07:39:01.579172 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:39:01.579630 13526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: Neither blocks nor log segments found. Creating new log.
I20260926 07:39:01.580912 13526 log.cc:824] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:39:01.584229 13526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: No bootstrap required, opened a new log
I20260926 07:39:01.582757 13353 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e445b57f42e54fc9bedc6237bbe24ad5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37364
I20260926 07:39:01.595278 13526 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.595731 13526 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcce89b49e414fcc8c711210cceed0dd, State: Initialized, Role: FOLLOWER
I20260926 07:39:01.596279 13526 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.597258 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.597594 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.598555 13526 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:39:01.608574 13378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 4
W20260926 07:39:01.608908 13544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:39:01.609057 13544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:39:01.606423 13459 master_runner.cc:186] Error getting master registration for 127.22.221.126:46037: OK, instance_id { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" instance_seqno: 1790408340455303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:39:01.618616 13459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:39:01.626499 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.609243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37106)
0926 07:39:01.609441 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.609465 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.609584 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.609994 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.609999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.610564 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.610791 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.610803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.612077 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.612085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.613736 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.613754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.615031 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.615037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.615293 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.616080 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.616104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.619218 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.623520 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.623528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.623532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.623960 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.624426 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.624432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.624436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.624624 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.625410 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.625416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.625610 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.625783 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.626094 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 07:39:01.626717 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.609342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.22.221.124:39709)
0926 07:39:01.609510 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.609516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.609538 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:39:01.609727 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.609733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.609832 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.609979 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.610952 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.611906 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.613940 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.614859 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.615016 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.619425 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.619432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.619441 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.619484 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.623383 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.624106 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.624114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.624120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.624278 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.624747 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.624752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.624756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.625265 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.625439 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.625903 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.626160 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:39:01.632264 13378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: Bootstrap starting.
I20260926 07:39:01.635789 13378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: Neither blocks nor log segments found. Creating new log.
I20260926 07:39:01.636967 13378 log.cc:824] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:39:01.640424 13378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: No bootstrap required, opened a new log
I20260926 07:39:01.646253 13378 raft_consensus.cc:348] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.646647 13378 raft_consensus.cc:729] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e445b57f42e54fc9bedc6237bbe24ad5, State: Initialized, Role: FOLLOWER
I20260926 07:39:01.647166 13378 consensus_queue.cc:260] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.648272 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.648689 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.649607 13378 sys_catalog.cc:564] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:39:01.649773 13353 raft_consensus.cc:482] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:39:01.650032 13353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:39:01.653843 13353 raft_consensus.cc:504] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.656500 13353 leader_election.cc:288] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [CANDIDATE]: Term 1 election: Requested vote from peers f2e35e0aacc041e0b3e20aa1e8aee626 (127.22.221.125:42455), dcce89b49e414fcc8c711210cceed0dd (127.22.221.124:39709)
I20260926 07:39:01.657346 13501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dcce89b49e414fcc8c711210cceed0dd"
I20260926 07:39:01.657871 13501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:39:01.661842 13501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e445b57f42e54fc9bedc6237bbe24ad5 in term 1.
I20260926 07:39:01.662600 13329 leader_election.cc:302] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcce89b49e414fcc8c711210cceed0dd, e445b57f42e54fc9bedc6237bbe24ad5; no voters:
I20260926 07:39:01.663161 13547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:39:01.664626 13311 master_runner.cc:186] Error getting master registration for 127.22.221.125:42455: OK, instance_id { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" instance_seqno: 1790408340956448 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:39:01.678109 13311 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:39:01.678732 13547 raft_consensus.cc:686] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [term 1 LEADER]: Becoming Leader. State: Replica: e445b57f42e54fc9bedc6237bbe24ad5, State: Running, Role: LEADER
I20260926 07:39:01.680490 13547 consensus_queue.cc:237] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
W20260926 07:39:01.684321 13558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:39:01.684484 13558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:39:01.684862 13452 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } attempt: 5
W20260926 07:39:01.694226 13332 tablet.cc:2369] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:39:01 dist-test-slave-qlq8 krb5kdc[13298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408340, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:39:01.700569 13561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e445b57f42e54fc9bedc6237bbe24ad5. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.701190 13561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:39:01.703065 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.685468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:39709 (local address 127.0.0.1:37112)
0926 07:39:01.685844 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:01.685864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:01.685979 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:01.686875 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:01.686881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:01.687425 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:01.687654 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.687668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.689161 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.689167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.690597 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:01.690612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.691809 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.691815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:01.692089 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.692821 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:01.692846 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:01.695755 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:01.700013 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.700018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.700022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.700528 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.700963 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:01.700967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:01.700970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:01.701331 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:01.702044 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:01.702049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:01.702218 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:39:01.702366 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 07:39:01.702684 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260926 07:39:01.704216 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.686211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.22.221.124:39709)
0926 07:39:01.686380 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.686386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.686404 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:39:01.686464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.686469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.686578 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.686684 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.687842 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.688973 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.690771 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.691487 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.691615 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.695975 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.695982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.695987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.696031 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.699820 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.700673 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.700678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.700682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.700836 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.701468 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.701472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.701475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.701835 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.701978 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.703591 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.703832 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:39:01.710510 13452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: Bootstrap starting.
I20260926 07:39:01.714735 13452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: Neither blocks nor log segments found. Creating new log.
I20260926 07:39:01.716300 13452 log.cc:824] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: Log is configured to *not* fsync() on all Append() calls
I20260926 07:39:01.717422 13566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:39:01.719723 13566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 07:39:01.723973 13480 tablet.cc:2369] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:39:01.727903 13452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: No bootstrap required, opened a new log
I20260926 07:39:01.755028 13452 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.755652 13452 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e35e0aacc041e0b3e20aa1e8aee626, State: Initialized, Role: FOLLOWER
I20260926 07:39:01.756315 13452 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } }
I20260926 07:39:01.761297 13501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [term 1 FOLLOWER]: Refusing update from remote peer e445b57f42e54fc9bedc6237bbe24ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:39:01.762874 13547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:39:01.768281 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.774170 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.777076 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.739550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.22.221.124:39709)
0926 07:39:01.739766 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.739772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.756274 (+ 16502us) server_negotiation.cc:407] Connection header received
0926 07:39:01.756620 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.756631 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.756713 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.756787 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.760466 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.761537 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.763677 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.764455 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.764567 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.768940 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.768948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.768953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.769005 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.773093 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.774514 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.774520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.774523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.774681 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.775171 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.775176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.775179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.775625 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.775878 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.776424 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.776667 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:39:01.779826 13452 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:39:01.779930 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626"
I20260926 07:39:01.780488 13427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:39:01.782979 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.734280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.22.221.125:42455)
0926 07:39:01.734440 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.734446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.752608 (+ 18162us) server_negotiation.cc:407] Connection header received
0926 07:39:01.753165 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.753177 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.753262 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.753340 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.756418 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.757789 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.769762 (+ 11973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.770582 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.770718 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.774563 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.774575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.774584 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.774640 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.778850 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.779713 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.779719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.779722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.779913 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.780528 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.780534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.780537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.780942 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.781226 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.782326 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.782588 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:39:01.783663 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:01.733822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.22.221.126:46037)
0926 07:39:01.734041 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:01.734050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:01.739700 (+ 5650us) server_negotiation.cc:407] Connection header received
0926 07:39:01.740268 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:01.740279 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:01.740395 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:01.740503 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:01.751728 (+ 11225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.757997 (+ 6269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.760303 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:01.760958 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:01.761060 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:01.771428 (+ 10368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:01.771437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:01.771442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:01.771492 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:01.775089 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.780975 (+ 5886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.780985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.780992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.781281 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:01.781741 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:01.781746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:01.781749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:01.782125 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:01.782391 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:01.782924 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 07:39:01.783364 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:39:01.786340 13427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e445b57f42e54fc9bedc6237bbe24ad5 in term 1.
I20260926 07:39:01.787216 13426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [term 1 FOLLOWER]: Refusing update from remote peer e445b57f42e54fc9bedc6237bbe24ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:39:01.789240 13547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:39:01.791698 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e445b57f42e54fc9bedc6237bbe24ad5. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.792004 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.806720 13570 mvcc.cc:205] Tried to move back new op lower bound from 7333512567837282304 to 7333512567535624192. Current Snapshot: MvccSnapshot[applied={T|T < 7333512567837282304}]
I20260926 07:39:01.812122 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.812537 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:39:01.819975 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.820344 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.823906 13566 catalog_manager.cc:1347] Generated new cluster ID: 398acb8a277f40eca0559f690afe1036
I20260926 07:39:01.824044 13566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:39:01.828023 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e445b57f42e54fc9bedc6237bbe24ad5. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.828375 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.831426 13561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.831707 13561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:39:01.850051 13566 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:39:01.852628 13566 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:39:01.860652 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e445b57f42e54fc9bedc6237bbe24ad5" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 46037 } } peers { permanent_uuid: "f2e35e0aacc041e0b3e20aa1e8aee626" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 42455 } } peers { permanent_uuid: "dcce89b49e414fcc8c711210cceed0dd" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 39709 } } }
I20260926 07:39:01.860970 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:01.885962 13566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e445b57f42e54fc9bedc6237bbe24ad5: Generated new TSK 0
I20260926 07:39:01.892066 13566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:39:01.955581 13591 catalog_manager.cc:1259] Loaded cluster ID: 398acb8a277f40eca0559f690afe1036
I20260926 07:39:01.955696 13591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: loading cluster ID for follower catalog manager: success
I20260926 07:39:01.971588 13591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: acquiring CA information for follower catalog manager: success
I20260926 07:39:01.974303 13591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2e35e0aacc041e0b3e20aa1e8aee626: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:39:02.066375 13530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.047812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.22.221.126:46037)
0926 07:39:02.048039 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.048046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.063950 (+ 15904us) server_negotiation.cc:407] Connection header received
0926 07:39:02.064281 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.064287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.064419 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.064508 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.066036 (+ 1528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37382: BlockingRecv error: recv got EOF from 127.0.0.1:37382 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 07:39:02.069387 13456 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.054049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.22.221.125:42455)
0926 07:39:02.054279 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.054291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.067367 (+ 13076us) server_negotiation.cc:407] Connection header received
0926 07:39:02.067656 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.067665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.067770 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.067857 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.069076 (+ 1219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37984: BlockingRecv error: recv got EOF from 127.0.0.1:37984 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:39:02.072410 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.064125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.22.221.124:39709)
0926 07:39:02.064354 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.064368 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.070692 (+ 6324us) server_negotiation.cc:407] Connection header received
0926 07:39:02.070983 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.070992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.071093 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.071171 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.072161 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37126: BlockingRecv error: recv got EOF from 127.0.0.1:37126 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:39:02.119858 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.100706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.22.221.125:42455)
0926 07:39:02.100913 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.100919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.111901 (+ 10982us) server_negotiation.cc:407] Connection header received
0926 07:39:02.112263 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.112270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.112494 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.112637 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.113377 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.114564 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.116704 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.117381 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.117483 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.118259 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.118995 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.119115 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.119405 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.119513 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:39:02.163566 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.160778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.22.221.124:39709)
0926 07:39:02.161022 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.161031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.161546 (+ 515us) server_negotiation.cc:407] Connection header received
0926 07:39:02.161917 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.161924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.162069 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.162150 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.163309 (+ 1159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37142: BlockingRecv error: recv got EOF from 127.0.0.1:37142 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:39:02.216511 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.201232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.22.221.125:42455)
0926 07:39:02.201491 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.201498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.209493 (+ 7995us) server_negotiation.cc:407] Connection header received
0926 07:39:02.210045 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.210052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.210148 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.210219 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.210917 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.212118 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.214167 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.214871 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.214985 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.215641 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.215793 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.215860 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.216102 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.216193 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260926 07:39:02.281023 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.265398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.22.221.124:39709)
0926 07:39:02.265618 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.265625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.279492 (+ 13867us) server_negotiation.cc:407] Connection header received
0926 07:39:02.279742 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.279748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.279838 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.279962 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.280821 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37144: BlockingRecv error: recv got EOF from 127.0.0.1:37144 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:39:02.294786 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.289042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.22.221.125:42455)
0926 07:39:02.289334 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.289343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.289900 (+ 557us) server_negotiation.cc:407] Connection header received
0926 07:39:02.290124 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.290131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.290214 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.290313 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.290752 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.291624 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.292958 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.293443 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.293593 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.293994 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.294137 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.294236 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.294394 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.294478 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 07:39:02.306906 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.304669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.22.221.124:39709)
0926 07:39:02.304871 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.304877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.305525 (+ 648us) server_negotiation.cc:407] Connection header received
0926 07:39:02.305746 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.305752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.305839 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.305950 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.306712 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37154: BlockingRecv error: recv got EOF from 127.0.0.1:37154 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:39:02.320667 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.315022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.22.221.125:42455)
0926 07:39:02.315312 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.315320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.315560 (+ 240us) server_negotiation.cc:407] Connection header received
0926 07:39:02.315775 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.315781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.315866 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.315964 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.316367 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.317147 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.318693 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.319353 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.319506 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.319876 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.319987 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.320090 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.320233 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.320315 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260926 07:39:02.333096 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.326894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.22.221.124:39709)
0926 07:39:02.327277 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.327286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.331729 (+ 4443us) server_negotiation.cc:407] Connection header received
0926 07:39:02.331941 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.331945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.332017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.332125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.332848 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37168: BlockingRecv error: recv got EOF from 127.0.0.1:37168 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
I20260926 07:39:02.349509 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.344024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.22.221.125:42455)
0926 07:39:02.344256 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.344265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.344737 (+ 472us) server_negotiation.cc:407] Connection header received
0926 07:39:02.344952 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.344958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.345038 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.345136 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.345542 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.346310 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.347733 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.348176 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.348313 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.348718 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.348833 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.348940 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.349085 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.349171 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 07:39:02.363426 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.361089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.22.221.124:39709)
0926 07:39:02.361334 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.361343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.361897 (+ 554us) server_negotiation.cc:407] Connection header received
0926 07:39:02.362109 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.362115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.362207 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.362315 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.363119 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37176: BlockingRecv error: recv got EOF from 127.0.0.1:37176 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 07:39:02.389549 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.383644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.22.221.125:42455)
0926 07:39:02.383879 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.383888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.384446 (+ 558us) server_negotiation.cc:407] Connection header received
0926 07:39:02.384730 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.384736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.384824 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.384927 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.385356 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.386414 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.387839 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.388236 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.388375 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.388784 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.388896 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.388999 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.389157 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.389246 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 07:39:02.397753 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.395596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.22.221.124:39709)
0926 07:39:02.395831 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.395840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.396368 (+ 528us) server_negotiation.cc:407] Connection header received
0926 07:39:02.396567 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.396573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.396671 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.396778 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.397525 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37184: BlockingRecv error: recv got EOF from 127.0.0.1:37184 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:39:02.439740 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.433424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.22.221.125:42455)
0926 07:39:02.433627 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.433633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.434000 (+ 367us) server_negotiation.cc:407] Connection header received
0926 07:39:02.434183 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.434190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.434282 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.434411 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.434809 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.436110 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.437496 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.438254 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.438402 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.438812 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.438954 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.439062 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.439188 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.439308 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 07:39:02.452745 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.450493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.22.221.124:39709)
0926 07:39:02.450752 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.450762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.451350 (+ 588us) server_negotiation.cc:407] Connection header received
0926 07:39:02.451557 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.451569 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.451641 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.451748 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.452521 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37190: BlockingRecv error: recv got EOF from 127.0.0.1:37190 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:39:02.526424 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.521018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.22.221.125:42455)
0926 07:39:02.521233 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.521242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.521602 (+ 360us) server_negotiation.cc:407] Connection header received
0926 07:39:02.521853 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.521858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.521930 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.522042 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.522448 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.523327 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.524656 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.525143 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.525317 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.525705 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.525811 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.525908 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.526057 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.526136 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 07:39:02.535116 13529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:02.533102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.22.221.124:39709)
0926 07:39:02.533315 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.533324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.533636 (+ 312us) server_negotiation.cc:407] Connection header received
0926 07:39:02.533784 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.533790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.533906 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.534016 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:02.534847 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37192: BlockingRecv error: recv got EOF from 127.0.0.1:37192 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:39:02.611109 13544 catalog_manager.cc:1259] Loaded cluster ID: 398acb8a277f40eca0559f690afe1036
I20260926 07:39:02.611212 13544 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: loading cluster ID for follower catalog manager: success
I20260926 07:39:02.613803 13544 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: acquiring CA information for follower catalog manager: success
I20260926 07:39:02.615686 13544 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcce89b49e414fcc8c711210cceed0dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:39:02.671923 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.666482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.22.221.125:42455)
0926 07:39:02.666836 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.666845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.667023 (+ 178us) server_negotiation.cc:407] Connection header received
0926 07:39:02.667175 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.667180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.667276 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.667363 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.667859 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.668580 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.669996 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.670476 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.670603 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.671043 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.671157 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.671258 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.671520 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.671591 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260926 07:39:02.683837 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.678007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.22.221.124:39709)
0926 07:39:02.678249 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.678254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.678505 (+ 251us) server_negotiation.cc:407] Connection header received
0926 07:39:02.678667 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.678672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.678845 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.678944 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.679349 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.680118 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.681482 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.681905 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.682001 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.682451 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.683124 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.683305 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.683436 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.683508 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 07:39:02.698930 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.690974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.22.221.125:42455)
0926 07:39:02.691213 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.691219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.692385 (+ 1166us) server_negotiation.cc:407] Connection header received
0926 07:39:02.692633 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.692642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.692751 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.692852 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.693580 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.694373 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.696753 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.697342 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.697491 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.697989 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.698131 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.698244 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.698522 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.698625 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:39:02.700601 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.692301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.22.221.124:39709)
0926 07:39:02.692501 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.692507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.693332 (+ 825us) server_negotiation.cc:407] Connection header received
0926 07:39:02.693533 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.693543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.693640 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.693746 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.694503 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.695707 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.698254 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.698912 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.699052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.699784 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.699892 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.699946 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.700154 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.700242 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:39:02.701040 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:02.690707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.22.221.126:46037)
0926 07:39:02.691213 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:02.691220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:02.691915 (+ 695us) server_negotiation.cc:407] Connection header received
0926 07:39:02.692190 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:02.692200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:02.692560 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:02.692793 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:02.693773 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.694839 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.697970 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:02.698639 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:02.698770 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:02.699583 (+ 813us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:02.700048 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:02.700153 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:02.700638 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 07:39:02.700751 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":254,"threads_started":1}
I20260926 07:39:02.707166 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13311
I20260926 07:39:02.708190 13373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:39:02.820509 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13311
I20260926 07:39:02.842635 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13385
I20260926 07:39:02.843966 13447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:39:02.953701 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13385
I20260926 07:39:02.975468 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13459
I20260926 07:39:02.976562 13521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:39:03.085597 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13459
2026-09-26T07:39:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4461 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:39:03 dist-test-slave-qlq8 krb5kdc[13705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:39:03 dist-test-slave-qlq8 krb5kdc[13705](info): set up 2 sockets
Sep 26 07:39:03 dist-test-slave-qlq8 krb5kdc[13705](info): commencing operation
krb5kdc: starting...
W20260926 07:39:04.204030 23413 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/test-user.keytab.
Sep 26 07:39:04 dist-test-slave-qlq8 krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
Entry for principal kudu/127.22.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab.
I20260926 07:39:04.469029 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35561
--webserver_interface=127.22.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35561,127.22.221.125:43401,127.22.221.124:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:39:04.803371 13718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:39:04.803709 13718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:39:04.803771 13718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:39:04.803855 13718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:39:04.811368 13718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:39:04.811474 13718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:39:04.811532 13718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:39:04.811584 13718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:39:04.811647 13718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:39:04.825448 13718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35561,127.22.221.125:43401,127.22.221.124:35391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.126.keytab
--principal=kudu/127.22.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.126:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:39:04.826797 13718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:39:04.828198 13718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:39:04.840137 13723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:04.840065 13724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:04.840032 13726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:39:04.841408 13718 server_base.cc:1034] running on GCE node
I20260926 07:39:04.842217 13718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:39:04.843696 13718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:39:04.844957 13718 hybrid_clock.cc:648] HybridClock initialized: now 1790408344844882 us; error 42 us; skew 500 ppm
I20260926 07:39:04.845322 13718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:39:04 dist-test-slave-qlq8 krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:39:04.848704 13718 init.cc:377] Logged in from keytab as kudu/127.22.221.126@KRBTEST.COM (short username kudu)
I20260926 07:39:04.850296 13718 webserver.cc:460] Webserver started at http://127.22.221.126:44479/ using document root <none> and password file <none>
I20260926 07:39:04.851066 13718 fs_manager.cc:359] Metadata directory not provided
I20260926 07:39:04.851270 13718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:39:04.851614 13718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:39:04.854552 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data/instance:
uuid: "af66424aaccb4309a21b975e149ea2a0"
format_stamp: "Formatted at 2026-09-26 07:39:04 on dist-test-slave-qlq8"
server_key: "f88dc2dc8326bf55d5ee786b94abbe41"
server_key_iv: "89b26d200fb312bdd6481da6a8d96a8a"
server_key_version: "encryptionkey@0"
I20260926 07:39:04.855428 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal/instance:
uuid: "af66424aaccb4309a21b975e149ea2a0"
format_stamp: "Formatted at 2026-09-26 07:39:04 on dist-test-slave-qlq8"
server_key: "f88dc2dc8326bf55d5ee786b94abbe41"
server_key_iv: "89b26d200fb312bdd6481da6a8d96a8a"
server_key_version: "encryptionkey@0"
I20260926 07:39:04.860523 13718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:39:04.863889 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:39:04.865249 13718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:39:04.865434 13718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal
uuid: "af66424aaccb4309a21b975e149ea2a0"
format_stamp: "Formatted at 2026-09-26 07:39:04 on dist-test-slave-qlq8"
server_key: "f88dc2dc8326bf55d5ee786b94abbe41"
server_key_iv: "89b26d200fb312bdd6481da6a8d96a8a"
server_key_version: "encryptionkey@0"
I20260926 07:39:04.865654 13718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:39:04.887900 13718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:39:04.892372 13718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:39:04.892597 13718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:39:04.911520 13718 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.126:35561
I20260926 07:39:04.911553 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.126:35561 every 8 connection(s)
I20260926 07:39:04.913337 13718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/data/info.pb
I20260926 07:39:04.917053 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:39:04.918119 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13718
I20260926 07:39:04.918324 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-0/wal/instance
I20260926 07:39:04.918889 23413 external_mini_cluster.cc:1428] Setting key d2a7e8f6a90c957fffc45241be81946b
I20260926 07:39:04.927435 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:04 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.125@KRBTEST.COM" created.
I20260926 07:39:04.954245 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:04.932642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.22.221.126:35561)
0926 07:39:04.933263 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:04.933269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:04.933312 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:39:04.935738 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:04.935744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:04.935993 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:04.936150 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:04.937771 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:04.938941 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:04.941210 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:04.941888 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:04.942087 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:04.946250 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:04.946261 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:04.946281 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:04.946326 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:04.949796 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:04.950617 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:04.950623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:04.950626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:04.950766 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:04.951179 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:04.951182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:04.951184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:04.951971 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:04.952158 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:04.952815 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 07:39:04.953147 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":458,"thread_start_us":307,"threads_started":1}
I20260926 07:39:04.954288 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:04.931551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35561 (local address 127.0.0.1:53842)
0926 07:39:04.932414 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:04.932464 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:04.935341 (+ 2877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:04.936253 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:04.936263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:04.937063 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:04.937553 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:04.937578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:04.939107 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:04.939115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:04.941032 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:04.941049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:04.942191 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:04.942200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:04.942435 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:04.943179 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:04.943213 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:04.945997 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:04.950000 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:04.950015 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:04.950030 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:04.950482 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:04.950865 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:04.950868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:04.950870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:04.951073 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:04.952139 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:04.952147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:04.952403 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 07:39:04.952686 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 07:39:04.953353 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":106,"thread_start_us":318,"threads_started":1}
I20260926 07:39:04.957355 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:04.958731 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:04.958215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44640)
0926 07:39:04.958431 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:04.958570 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 07:39:04.959093 13737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
I20260926 07:39:04.960788 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:04.960371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44652)
0926 07:39:04.960563 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:04.960657 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 07:39:04.961001 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:43401: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab.
Entry for principal kudu/127.22.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab.
I20260926 07:39:04.984539 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:43401
--webserver_interface=127.22.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab
--principal=kudu/127.22.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35561,127.22.221.125:43401,127.22.221.124:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:39:05.010349 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } attempt: 1
I20260926 07:39:05.011555 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.011053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44664)
0926 07:39:05.011297 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.011411 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:39:05.012686 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.012237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44678)
0926 07:39:05.012392 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.012514 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:39:05.012866 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:43401: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
I20260926 07:39:05.081161 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } attempt: 2
I20260926 07:39:05.082438 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.081920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44690)
0926 07:39:05.082200 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.082292 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 07:39:05.083626 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.083153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44698)
0926 07:39:05.083370 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.083484 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 07:39:05.083825 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:43401: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
I20260926 07:39:05.175093 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } attempt: 3
I20260926 07:39:05.176381 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.175940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44700)
0926 07:39:05.176146 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.176245 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:39:05.177454 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.177077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44708)
0926 07:39:05.177227 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.177310 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:39:05.177644 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:43401: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
W20260926 07:39:05.310221 13792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:39:05.310629 13792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:39:05.310761 13792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:39:05.310904 13792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:39:05.318212 13792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:39:05.318356 13792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:39:05.318463 13792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:39:05.318562 13792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:39:05.318651 13792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:39:05.320937 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } attempt: 4
I20260926 07:39:05.322108 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.321594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44724)
0926 07:39:05.321890 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.321980 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 07:39:05.323086 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.322736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44730)
0926 07:39:05.322880 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.322969 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:39:05.323302 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.125:43401: Network error: Client connection negotiation failed: client connection to 127.22.221.125:43401: connect: Connection refused (error 111)
I20260926 07:39:05.329058 13792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35561,127.22.221.125:43401,127.22.221.124:35391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.125.keytab
--principal=kudu/127.22.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.125:43401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:39:05.330425 13792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:39:05.331854 13792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:39:05.344621 13797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:05.344661 13800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:05.344708 13798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:39:05.348807 13792 server_base.cc:1034] running on GCE node
I20260926 07:39:05.349725 13792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:39:05.351524 13792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:39:05.352800 13792 hybrid_clock.cc:648] HybridClock initialized: now 1790408345352719 us; error 50 us; skew 500 ppm
I20260926 07:39:05.353174 13792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:39:05.356463 13792 init.cc:377] Logged in from keytab as kudu/127.22.221.125@KRBTEST.COM (short username kudu)
I20260926 07:39:05.358119 13792 webserver.cc:460] Webserver started at http://127.22.221.125:41617/ using document root <none> and password file <none>
I20260926 07:39:05.358865 13792 fs_manager.cc:359] Metadata directory not provided
I20260926 07:39:05.358997 13792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:39:05.359400 13792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:39:05.362409 13792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data/instance:
uuid: "10ff902d56144fb8889565663d5b14f9"
format_stamp: "Formatted at 2026-09-26 07:39:05 on dist-test-slave-qlq8"
server_key: "0bc0622e94ad92f9144a7af469b3e70d"
server_key_iv: "f629a9bc98eb319fa19ce808995c520a"
server_key_version: "encryptionkey@0"
I20260926 07:39:05.363291 13792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal/instance:
uuid: "10ff902d56144fb8889565663d5b14f9"
format_stamp: "Formatted at 2026-09-26 07:39:05 on dist-test-slave-qlq8"
server_key: "0bc0622e94ad92f9144a7af469b3e70d"
server_key_iv: "f629a9bc98eb319fa19ce808995c520a"
server_key_version: "encryptionkey@0"
I20260926 07:39:05.368427 13792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 07:39:05.372018 13807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:39:05.373342 13792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:39:05.373528 13792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal
uuid: "10ff902d56144fb8889565663d5b14f9"
format_stamp: "Formatted at 2026-09-26 07:39:05 on dist-test-slave-qlq8"
server_key: "0bc0622e94ad92f9144a7af469b3e70d"
server_key_iv: "f629a9bc98eb319fa19ce808995c520a"
server_key_version: "encryptionkey@0"
I20260926 07:39:05.373744 13792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:39:05.395146 13792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:39:05.399194 13792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:39:05.399523 13792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:39:05.418174 13792 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.125:43401
I20260926 07:39:05.418195 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.125:43401 every 8 connection(s)
I20260926 07:39:05.419564 13792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/data/info.pb
I20260926 07:39:05.420507 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13792
I20260926 07:39:05.420729 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-1/wal/instance
I20260926 07:39:05.421221 23413 external_mini_cluster.cc:1428] Setting key 21ea4804be87b8d33e6050de4399cd27
I20260926 07:39:05.423565 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:39:05.429994 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.221.124@KRBTEST.COM" created.
I20260926 07:39:05.450455 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.432929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.22.221.126:35561)
0926 07:39:05.433121 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.433125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.434342 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 07:39:05.436270 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.436276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.436371 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.436481 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.437955 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.438649 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.440311 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.440737 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.440894 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.444597 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.444602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.444605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.444638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.447441 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.448173 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.448178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.448180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.448293 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.448720 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.448725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.448730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.449006 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.449171 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.449870 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.450105 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:39:05.451510 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.433433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35561 (local address 127.0.0.1:53858)
0926 07:39:05.434179 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.434220 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:05.435991 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:05.436714 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:05.436722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:05.437242 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:05.437781 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.437798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.438819 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.438823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.440149 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.440161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.440970 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.440977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.441218 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.441947 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:05.441970 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:05.444352 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:05.447578 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.447587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.447604 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.448038 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.448402 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.448408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.448412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.448596 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.449144 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:05.449152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:05.449498 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 07:39:05.449767 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 07:39:05.450291 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":311,"threads_started":1}
I20260926 07:39:05.453236 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:39:05.470265 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.454032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44746)
0926 07:39:05.454230 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.454251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:05.454361 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:05.455323 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:05.455330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:05.455813 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:05.455963 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.455971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.458024 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.458028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.458840 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.458850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.460065 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.460070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.460228 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.460906 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:05.460924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:05.463029 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:05.467487 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.467491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.467494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.467873 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.468307 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.468311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.468313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.468478 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.469280 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:05.469284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:05.469485 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 07:39:05.469664 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 07:39:05.469994 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3,"spinlock_wait_cycles":17280}
I20260926 07:39:05.470743 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.454225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.22.221.125:43401)
0926 07:39:05.454862 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.454867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.454891 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:39:05.454939 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.454944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.455120 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.455275 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.456139 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.457890 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.458978 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.459932 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.460074 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.463273 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.463290 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.463312 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.463361 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.467321 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.468013 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.468031 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.468037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.468192 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.468592 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.468599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.468605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.469132 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.469345 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.469837 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.470181 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":496,"thread_start_us":369,"threads_started":1}
I20260926 07:39:05.473526 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:05.474632 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.474193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47822)
0926 07:39:05.474342 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.474479 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:39:05.474932 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.476608 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.476242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47826)
0926 07:39:05.476410 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.476496 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:39:05.476873 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab.
Entry for principal kudu/127.22.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab.
I20260926 07:39:05.483750 23413 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
/tmp/dist-test-taskdo7xD1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:35391
--webserver_interface=127.22.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab
--principal=kudu/127.22.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.221.84:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.221.126:35561,127.22.221.125:43401,127.22.221.124:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:39:05.526292 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 1
I20260926 07:39:05.527699 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.527129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47830)
0926 07:39:05.527430 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.527543 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:39:05.528702 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.528371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47846)
0926 07:39:05.528517 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.528599 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:39:05.528941 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.597194 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 2
I20260926 07:39:05.598637 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.598013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47848)
0926 07:39:05.598289 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.598424 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 07:39:05.600041 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.599551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47856)
0926 07:39:05.599745 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.599870 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:39:05.600342 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.622556 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } attempt: 5
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:39:05.636783 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.623567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.22.221.125:43401)
0926 07:39:05.623768 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.623776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.623800 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:39:05.623866 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.623879 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.623958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.624094 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.624888 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.625590 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.626759 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.627195 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.627363 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.630817 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.630842 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.630845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.630878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.633791 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.634660 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.634664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.634667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.634794 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.635343 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.635349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.635353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.635669 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.635810 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.636373 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.636554 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:39:05.637171 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.623360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44748)
0926 07:39:05.623568 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.623592 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:05.623733 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:05.624134 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:05.624139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:05.624546 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:05.624724 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.624732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.625734 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.625740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.626601 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.626612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.627395 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.627400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.627621 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.628266 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:05.628283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:05.630611 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:05.633944 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.633949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.633953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.634549 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.634920 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.634936 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.634942 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.635144 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.635806 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:05.635814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:05.636078 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 07:39:05.636274 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 07:39:05.636661 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:39:05.638298 13785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:05.639364 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.638947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47860)
0926 07:39:05.639114 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.639190 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:39:05.640291 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.639984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47868)
0926 07:39:05.640132 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.640197 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:39:05.640610 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.691579 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 3
I20260926 07:39:05.691866 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 1
I20260926 07:39:05.692838 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.692315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47874)
0926 07:39:05.692537 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.692643 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:39:05.693310 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.692538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47880)
0926 07:39:05.692749 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.692848 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 07:39:05.694043 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.693520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47882)
0926 07:39:05.693805 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.693896 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260926 07:39:05.694255 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.694285 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.693927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47894)
0926 07:39:05.694072 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.694159 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:39:05.694540 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.762799 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 2
I20260926 07:39:05.764048 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.763556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47910)
0926 07:39:05.763795 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.763894 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:39:05.765162 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.764800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47914)
0926 07:39:05.764970 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.765061 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:39:05.765434 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
W20260926 07:39:05.822309 13866 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:39:05.822682 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:39:05.822798 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:39:05.822932 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:39:05.830216 13866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:39:05.830329 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:39:05.830443 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:39:05.830524 13866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:39:05.830646 13866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:39:05.837502 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 4
I20260926 07:39:05.838769 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.838221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47918)
0926 07:39:05.838480 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.838589 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:39:05.839869 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.839506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47932)
0926 07:39:05.839655 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.839736 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:39:05.840161 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.842070 13866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.221.84:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.221.126:35561,127.22.221.125:43401,127.22.221.124:35391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/krb5kdc/kudu_127.22.221.124.keytab
--principal=kudu/127.22.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.221.124:35391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:39:05.843638 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:39:05.845052 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:39:05.856730 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:05.856817 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:39:05.856786 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:39:05.857618 13866 server_base.cc:1034] running on GCE node
I20260926 07:39:05.858525 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:39:05.860152 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:39:05.861478 13866 hybrid_clock.cc:648] HybridClock initialized: now 1790408345861376 us; error 66 us; skew 500 ppm
I20260926 07:39:05.861824 13866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:39:05.865115 13866 init.cc:377] Logged in from keytab as kudu/127.22.221.124@KRBTEST.COM (short username kudu)
I20260926 07:39:05.866720 13866 webserver.cc:460] Webserver started at http://127.22.221.124:45423/ using document root <none> and password file <none>
I20260926 07:39:05.867538 13866 fs_manager.cc:359] Metadata directory not provided
I20260926 07:39:05.867668 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:39:05.867986 13866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:39:05.871644 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 3
I20260926 07:39:05.871829 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data/instance:
uuid: "ae8a08f82820458cb25b2b38aa49b931"
format_stamp: "Formatted at 2026-09-26 07:39:05 on dist-test-slave-qlq8"
server_key: "fd605753c86dfef38c78c36d8a40ba7b"
server_key_iv: "05808208509b861d7d271562e906aabd"
server_key_version: "encryptionkey@0"
I20260926 07:39:05.872807 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal/instance:
uuid: "ae8a08f82820458cb25b2b38aa49b931"
format_stamp: "Formatted at 2026-09-26 07:39:05 on dist-test-slave-qlq8"
server_key: "fd605753c86dfef38c78c36d8a40ba7b"
server_key_iv: "05808208509b861d7d271562e906aabd"
server_key_version: "encryptionkey@0"
I20260926 07:39:05.872850 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.872248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47938)
0926 07:39:05.872472 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.872664 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 07:39:05.873961 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.873571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47946)
0926 07:39:05.873770 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.873849 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 07:39:05.874248 13785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.221.124:35391: Network error: Client connection negotiation failed: client connection to 127.22.221.124:35391: connect: Connection refused (error 111)
I20260926 07:39:05.878921 13866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 07:39:05.882843 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:39:05.884325 13866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:39:05.884544 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data,/tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal
uuid: "ae8a08f82820458cb25b2b38aa49b931"
format_stamp: "Formatted at 2026-09-26 07:39:05 on dist-test-slave-qlq8"
server_key: "fd605753c86dfef38c78c36d8a40ba7b"
server_key_iv: "05808208509b861d7d271562e906aabd"
server_key_version: "encryptionkey@0"
I20260926 07:39:05.884790 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:39:05.905704 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:39:05.909971 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:39:05.910260 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:39:05.929795 13866 rpc_server.cc:307] RPC server started. Bound to: 127.22.221.124:35391
I20260926 07:39:05.929765 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.221.124:35391 every 8 connection(s)
I20260926 07:39:05.931375 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/data/info.pb
I20260926 07:39:05.935513 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:39:05.940181 23413 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu as pid 13866
I20260926 07:39:05.940379 23413 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdo7xD1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407989013554-23413-0/minicluster-data/master-2/wal/instance
I20260926 07:39:05.940800 23413 external_mini_cluster.cc:1428] Setting key d74a7d79e247d4d9a652e947a06a9051
I20260926 07:39:05.942911 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:39:05.963835 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.945626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.22.221.126:35561)
0926 07:39:05.945921 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.945929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.946417 (+ 488us) server_negotiation.cc:407] Connection header received
0926 07:39:05.946733 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.946740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.946821 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.946920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.948609 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.949648 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.951943 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.952636 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.952787 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.957140 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.957148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.957153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.957199 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.960102 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.961047 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.961051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.961053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.961185 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.961759 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.961764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.961766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.962187 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.962359 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.963321 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.963509 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 07:39:05.965307 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.945532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.126:35561 (local address 127.0.0.1:53870)
0926 07:39:05.946196 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.946259 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:05.946570 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:05.947035 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:05.947041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:05.947916 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:05.948436 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.948452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.949829 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.949833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.951783 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.951799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.952956 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.952966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.953207 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.954018 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:05.954050 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:05.956908 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:05.960235 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.960246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.960258 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.960904 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.961292 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.961296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.961299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.961576 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.962360 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:05.962366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:05.962843 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 07:39:05.963147 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 07:39:05.963749 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":302,"threads_started":1}
I20260926 07:39:05.965412 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.942531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.22.221.124:35391)
0926 07:39:05.943359 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.943370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.943418 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:39:05.945309 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.945339 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.945928 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.946347 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.948206 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.949934 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.951933 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.952807 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.953043 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.957772 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.957780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.957821 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.957861 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.961305 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.962540 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.962547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.962550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.962713 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.963116 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.963123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.963125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.963573 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.963909 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.964745 (+ 836us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.965040 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":355,"threads_started":1}
I20260926 07:39:05.967839 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.126@KRBTEST.COM
I20260926 07:39:05.982290 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.968706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.125:43401 (local address 127.0.0.1:44754)
0926 07:39:05.968850 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.968869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:05.968969 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:05.969381 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:05.969385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:05.969767 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:05.969908 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.969916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.970973 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.970980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.971977 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.971988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.972710 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.972715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.972893 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.973496 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:05.973513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:05.975868 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:05.979857 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.979868 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.979871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.980216 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.980628 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.980631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.980634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.980771 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.981340 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:05.981344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:05.981498 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 07:39:05.981651 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:39:05.981925 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:39:05.982316 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.968788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.22.221.125:43401)
0926 07:39:05.968995 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.969004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.969025 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:39:05.969114 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.969120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.969243 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.969364 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.970054 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.970839 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.972122 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.972557 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.972687 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.976068 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.976073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.976076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.976109 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.979724 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.980335 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.980341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.980345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.980511 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.980875 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.980882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.980894 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.981225 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.981372 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.981734 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.981948 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:39:05.983727 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:39:05.984350 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.968492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.22.221.126:35561)
0926 07:39:05.968700 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.968706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.969040 (+ 334us) server_negotiation.cc:407] Connection header received
0926 07:39:05.969242 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.969247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.969338 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.969449 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.970631 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.971428 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.972651 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.973096 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.973228 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.977502 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.977509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.977515 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.977583 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.980966 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.981536 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.981542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.981546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.981678 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.982043 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.982048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.982052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.982322 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.982624 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.983217 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.983497 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.124@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
Sep 26 07:39:05 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.221.125@KRBTEST.COM
I20260926 07:39:05.998281 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.984642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.22.221.124:35391)
0926 07:39:05.984799 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.984805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.984821 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:39:05.984907 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.984911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.984960 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.985033 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.985834 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.986824 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.987998 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.988406 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.988539 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.991895 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.991899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.991902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.991936 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.995369 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.996264 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.996268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.996271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.996410 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.996820 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.996827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.996831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.997121 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:05.997276 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:05.997627 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:39:05.997854 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":51}
I20260926 07:39:05.998325 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.984480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47964)
0926 07:39:05.984659 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:05.984678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:05.984781 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:05.985097 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:05.985101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:05.985531 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:05.985705 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.985717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.986936 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.986944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.987864 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:05.987876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.988548 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.988553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:05.988816 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.989453 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:05.989473 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:05.991704 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:05.995543 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.995548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.995550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.996131 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.996530 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:05.996573 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:05.996577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:05.996711 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:05.997238 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:05.997243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:05.997397 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 07:39:05.997553 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 07:39:05.998000 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":141}
I20260926 07:39:06.001858 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:05.986092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.22.221.125:43401)
0926 07:39:05.986249 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:05.986254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:05.986271 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:39:05.986379 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:05.986383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:05.986451 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:05.986537 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:05.987766 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.988453 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.989946 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:05.990645 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:05.990782 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:05.995762 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:05.995769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:05.995774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:05.995816 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:05.998781 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:05.999802 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:05.999808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:05.999811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:05.999993 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.000488 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.000491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.000493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.000767 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:06.000963 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.001350 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.001568 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:39:06.005244 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: Bootstrap starting.
I20260926 07:39:06.007097 23413 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:39:06.008873 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: Neither blocks nor log segments found. Creating new log.
I20260926 07:39:06.010022 13933 log.cc:824] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: Log is configured to *not* fsync() on all Append() calls
I20260926 07:39:06.009724 13760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "af66424aaccb4309a21b975e149ea2a0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884
I20260926 07:39:06.013028 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: No bootstrap required, opened a new log
I20260926 07:39:06.020326 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.020701 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8a08f82820458cb25b2b38aa49b931, State: Initialized, Role: FOLLOWER
I20260926 07:39:06.021159 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.022034 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.022373 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.023444 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:39:06.033597 13950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:39:06.033753 13950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:39:06.031157 13866 master_runner.cc:186] Error getting master registration for 127.22.221.126:35561: OK, instance_id { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" instance_seqno: 1790408344906159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:39:06.042510 13866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:39:06.051499 13785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 4
Sep 26 07:39:06 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408344, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.126@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:39:06.068315 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.052231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:47978)
0926 07:39:06.052456 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:06.052479 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:06.052609 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:06.052994 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:06.052999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:06.053523 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:06.053737 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:06.053748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.055371 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.055376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:06.056840 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:06.056856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.057866 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.057873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:06.058078 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.058785 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:06.058802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:06.061238 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:06.065363 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:06.065371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:06.065376 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:06.065923 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:06.066340 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:06.066345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:06.066348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:06.066513 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:06.067260 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:06.067265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:06.067445 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 07:39:06.067616 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 07:39:06.067900 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:39:06.068377 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.052407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.22.221.124:35391)
0926 07:39:06.052649 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.052655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.052677 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:39:06.052744 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.052751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.052831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.052945 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.053904 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.055154 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.057002 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.057708 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.057845 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.061416 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:06.061423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:06.061429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:06.061470 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:06.065212 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.066062 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.066067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.066071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.066221 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.066630 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.066635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.066639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.067103 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:06.067287 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.067750 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.068007 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:39:06.074882 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: Bootstrap starting.
I20260926 07:39:06.079409 13785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: Neither blocks nor log segments found. Creating new log.
I20260926 07:39:06.080871 13785 log.cc:824] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:39:06.084553 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: No bootstrap required, opened a new log
I20260926 07:39:06.092892 13785 raft_consensus.cc:348] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.093338 13785 raft_consensus.cc:729] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af66424aaccb4309a21b975e149ea2a0, State: Initialized, Role: FOLLOWER
I20260926 07:39:06.093904 13785 consensus_queue.cc:260] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.094911 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.095345 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.096278 13785 sys_catalog.cc:564] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:39:06.096454 13760 raft_consensus.cc:482] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:39:06.096666 13760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:39:06.099862 13760 raft_consensus.cc:504] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.102288 13908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af66424aaccb4309a21b975e149ea2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ae8a08f82820458cb25b2b38aa49b931"
I20260926 07:39:06.102764 13908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:39:06.106189 13908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af66424aaccb4309a21b975e149ea2a0 in term 1.
I20260926 07:39:06.106806 13734 leader_election.cc:302] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae8a08f82820458cb25b2b38aa49b931, af66424aaccb4309a21b975e149ea2a0; no voters:
I20260926 07:39:06.107290 13953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:39:06.107290 13760 leader_election.cc:288] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [CANDIDATE]: Term 1 election: Requested vote from peers 10ff902d56144fb8889565663d5b14f9 (127.22.221.125:43401), ae8a08f82820458cb25b2b38aa49b931 (127.22.221.124:35391)
I20260926 07:39:06.134203 13953 raft_consensus.cc:686] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [term 1 LEADER]: Becoming Leader. State: Replica: af66424aaccb4309a21b975e149ea2a0, State: Running, Role: LEADER
I20260926 07:39:06.134933 13953 consensus_queue.cc:237] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.139358 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } attempt: 5
W20260926 07:39:06.145460 13739 tablet.cc:2369] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:39:06.146445 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af66424aaccb4309a21b975e149ea2a0. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.146914 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: This master's current role is: LEADER
W20260926 07:39:06.162724 13887 tablet.cc:2369] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:39:06 dist-test-slave-qlq8 krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.221.125@KRBTEST.COM for kudu/127.22.221.124@KRBTEST.COM
I20260926 07:39:06.157279 13718 master_runner.cc:186] Error getting master registration for 127.22.221.125:43401: OK, instance_id { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" instance_seqno: 1790408345412901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:39:06.170944 13718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:39:06.172994 13974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:39:06.173111 13974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:39:06.173243 13975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:39:06.175623 13975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:39:06.186450 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.131725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.22.221.126:35561)
0926 07:39:06.131937 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.131944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.133699 (+ 1755us) server_negotiation.cc:407] Connection header received
0926 07:39:06.138263 (+ 4564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.138272 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.138381 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.138469 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.141594 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.142677 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.145588 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.146418 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.146534 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.173864 (+ 27330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:06.173871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:06.173877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:06.173929 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:06.178807 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.180763 (+ 1956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.180769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.180773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.180963 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.182050 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.182059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.182064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.182474 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:06.182790 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.185491 (+ 2701us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.185773 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:39:06.188411 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.139988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.221.124:35391 (local address 127.0.0.1:48006)
0926 07:39:06.140387 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 07:39:06.140408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:39:06.140518 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:39:06.160693 (+ 20175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:39:06.160699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:39:06.161211 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:39:06.161426 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:06.161439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.163255 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.163262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:06.164749 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:39:06.164765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.165816 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.165822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:39:06.166094 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.166817 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:39:06.166839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:39:06.169723 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:39:06.180002 (+ 10279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:06.180008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:06.180012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:06.180578 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:06.181054 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:39:06.181060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:39:06.181066 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:39:06.181276 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:39:06.187335 (+ 6059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:39:06.187342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:39:06.187529 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 07:39:06.187672 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:39:06.188008 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260926 07:39:06.193094 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.140752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.22.221.124:35391)
0926 07:39:06.160302 (+ 19550us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.160309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.160336 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:39:06.160389 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.160395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.160477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.161583 (+ 1106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.161849 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.163077 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.164956 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.165638 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.169891 (+ 4253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.170146 (+ 255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:06.170155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:06.170162 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:06.170193 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:06.179818 (+ 9625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.180754 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.180759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.180764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.180928 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.186700 (+ 5772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.186706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.186713 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.187143 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:06.188693 (+ 1550us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.188819 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.189055 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19399,"thread_start_us":28661,"threads_started":1}
I20260926 07:39:06.197374 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.133852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.22.221.124:35391)
0926 07:39:06.134034 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.134039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.141393 (+ 7354us) server_negotiation.cc:407] Connection header received
0926 07:39:06.142559 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.142565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.142641 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.142711 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.158401 (+ 15690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.159909 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.174917 (+ 15008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.175649 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.175758 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.184374 (+ 8616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:06.184381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:06.184386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:06.184422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:06.188104 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.194285 (+ 6181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.194293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.194298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.194449 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.194872 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.194877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.194880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.195269 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:06.195513 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.196735 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.197005 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:39:06.199021 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.134841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.22.221.125:43401)
0926 07:39:06.135015 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.135021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.141938 (+ 6917us) server_negotiation.cc:407] Connection header received
0926 07:39:06.142922 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.142930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.143013 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.143083 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.166113 (+ 23030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.167660 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.179723 (+ 12063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.180491 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.180585 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.189363 (+ 8778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:39:06.189370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:39:06.189376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:39:06.189419 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:39:06.193150 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.193949 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.193957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.193964 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.194136 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:39:06.195169 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:39:06.195174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:39:06.195182 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:39:06.195589 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:39:06.195822 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.198189 (+ 2367us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.198426 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260926 07:39:06.201036 13908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [term 1 FOLLOWER]: Refusing update from remote peer af66424aaccb4309a21b975e149ea2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:39:06.202162 13961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:39:06.204249 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: Bootstrap starting.
I20260926 07:39:06.208266 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: Neither blocks nor log segments found. Creating new log.
I20260926 07:39:06.209654 13859 log.cc:824] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:39:06.226200 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: No bootstrap required, opened a new log
I20260926 07:39:06.236744 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af66424aaccb4309a21b975e149ea2a0. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.237144 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.238848 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.239385 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ff902d56144fb8889565663d5b14f9, State: Initialized, Role: FOLLOWER
I20260926 07:39:06.240002 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } }
I20260926 07:39:06.248971 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.249352 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: This master's current role is: LEADER
I20260926 07:39:06.252202 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.252473 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.253458 13975 catalog_manager.cc:1347] Generated new cluster ID: 681c42b8e7a4468e99755100569992ce
I20260926 07:39:06.253580 13975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:39:06.255151 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.255671 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.256763 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:39:06.256819 13834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af66424aaccb4309a21b975e149ea2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "10ff902d56144fb8889565663d5b14f9"
I20260926 07:39:06.257313 13834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:39:06.261545 13834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af66424aaccb4309a21b975e149ea2a0 in term 1.
I20260926 07:39:06.262784 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [term 1 FOLLOWER]: Refusing update from remote peer af66424aaccb4309a21b975e149ea2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:39:06.263895 13961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:39:06.299271 13975 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:39:06.301192 13975 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:39:06.307533 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af66424aaccb4309a21b975e149ea2a0. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.308719 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.308255 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.308939 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:39:06.310118 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af66424aaccb4309a21b975e149ea2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af66424aaccb4309a21b975e149ea2a0" member_type: VOTER last_known_addr { host: "127.22.221.126" port: 35561 } } peers { permanent_uuid: "10ff902d56144fb8889565663d5b14f9" member_type: VOTER last_known_addr { host: "127.22.221.125" port: 43401 } } peers { permanent_uuid: "ae8a08f82820458cb25b2b38aa49b931" member_type: VOTER last_known_addr { host: "127.22.221.124" port: 35391 } } }
I20260926 07:39:06.310443 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0 [sys.catalog]: This master's current role is: LEADER
I20260926 07:39:06.331161 13975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af66424aaccb4309a21b975e149ea2a0: Generated new TSK 0
I20260926 07:39:06.332304 13975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:39:06.372138 13999 catalog_manager.cc:1259] Loaded cluster ID: 681c42b8e7a4468e99755100569992ce
I20260926 07:39:06.372246 13999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: loading cluster ID for follower catalog manager: success
I20260926 07:39:06.375928 13999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: acquiring CA information for follower catalog manager: success
I20260926 07:39:06.378276 13999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10ff902d56144fb8889565663d5b14f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:39:06.489924 13789 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.482080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.22.221.126:35561)
0926 07:39:06.482274 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.482280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.487818 (+ 5538us) server_negotiation.cc:407] Connection header received
0926 07:39:06.488153 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.488159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.488249 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.488331 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.489644 (+ 1313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53890: BlockingRecv error: recv got EOF from 127.0.0.1:53890 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 07:39:06.491272 13863 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.481674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.22.221.125:43401)
0926 07:39:06.481914 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.481923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.489225 (+ 7302us) server_negotiation.cc:407] Connection header received
0926 07:39:06.489472 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.489477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.489562 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.489634 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.490783 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44778: BlockingRecv error: recv got EOF from 127.0.0.1:44778 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 07:39:06.496544 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.488003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.22.221.124:35391)
0926 07:39:06.488222 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.488228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.491748 (+ 3520us) server_negotiation.cc:407] Connection header received
0926 07:39:06.495045 (+ 3297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.495052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.495142 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.495217 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.496278 (+ 1061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48016: BlockingRecv error: recv got EOF from 127.0.0.1:48016 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 07:39:06.553246 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.533570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.22.221.125:43401)
0926 07:39:06.533780 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.533787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.540755 (+ 6968us) server_negotiation.cc:407] Connection header received
0926 07:39:06.541336 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.541343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.541623 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.541788 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.542571 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.547123 (+ 4552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.549629 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.550490 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.550609 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.551570 (+ 961us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.552342 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.552473 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.552756 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.552865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 07:39:06.603701 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.590040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.22.221.124:35391)
0926 07:39:06.590312 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.590328 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.601274 (+ 10946us) server_negotiation.cc:407] Connection header received
0926 07:39:06.601887 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.601895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.602038 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.602127 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.603433 (+ 1306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48032: BlockingRecv error: recv got EOF from 127.0.0.1:48032 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 07:39:06.660359 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.639218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.22.221.125:43401)
0926 07:39:06.639519 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.639526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.651941 (+ 12415us) server_negotiation.cc:407] Connection header received
0926 07:39:06.652222 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.652229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.652348 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.652426 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.653711 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.655103 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.657467 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.658267 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.658383 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.659273 (+ 890us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.659458 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.659563 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.659913 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.660013 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 07:39:06.705633 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.703616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.22.221.124:35391)
0926 07:39:06.703797 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.703805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.703841 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:39:06.704054 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.704061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.704165 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.704252 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.705389 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48034: BlockingRecv error: recv got EOF from 127.0.0.1:48034 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:39:06.722648 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.712219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.22.221.125:43401)
0926 07:39:06.712426 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.712433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.715871 (+ 3438us) server_negotiation.cc:407] Connection header received
0926 07:39:06.716106 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.716112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.716224 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.716336 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.716863 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.718066 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.720372 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.721093 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.721231 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.721757 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.721928 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.722024 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.722197 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.722290 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 07:39:06.733319 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.730626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.22.221.124:35391)
0926 07:39:06.730853 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.730860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.731766 (+ 906us) server_negotiation.cc:407] Connection header received
0926 07:39:06.731968 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.731975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.732071 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.732186 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.733086 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48042: BlockingRecv error: recv got EOF from 127.0.0.1:48042 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:39:06.748423 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.741052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.22.221.125:43401)
0926 07:39:06.741319 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.741327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.741982 (+ 655us) server_negotiation.cc:407] Connection header received
0926 07:39:06.742194 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.742200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.742313 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.742432 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.742903 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.744211 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.746133 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.746890 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.747022 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.747587 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.747730 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.747840 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.748017 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.748106 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260926 07:39:06.757962 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.756258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.22.221.124:35391)
0926 07:39:06.756491 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.756500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.756552 (+ 52us) server_negotiation.cc:407] Connection header received
0926 07:39:06.756631 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.756638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.756731 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.756835 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.757720 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48044: BlockingRecv error: recv got EOF from 127.0.0.1:48044 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 07:39:06.785490 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.777664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.22.221.125:43401)
0926 07:39:06.777884 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.777892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.778512 (+ 620us) server_negotiation.cc:407] Connection header received
0926 07:39:06.778775 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.778782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.778880 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.778997 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.779619 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.780821 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.783088 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.783866 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.784016 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.784573 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.784736 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.784839 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.785054 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.785140 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 07:39:06.797660 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.792935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.22.221.124:35391)
0926 07:39:06.793123 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.793129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.795725 (+ 2596us) server_negotiation.cc:407] Connection header received
0926 07:39:06.795943 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.795950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.796061 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.796175 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.797401 (+ 1226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48052: BlockingRecv error: recv got EOF from 127.0.0.1:48052 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:39:06.827006 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.819069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.22.221.125:43401)
0926 07:39:06.819334 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.819341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.819932 (+ 591us) server_negotiation.cc:407] Connection header received
0926 07:39:06.820202 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.820210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.820316 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.820410 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.820962 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.822218 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.824554 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.825331 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.825473 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.826074 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.826252 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.826345 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.826559 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.826655 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 07:39:06.837263 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.834969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.22.221.124:35391)
0926 07:39:06.835150 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.835158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.835709 (+ 551us) server_negotiation.cc:407] Connection header received
0926 07:39:06.835886 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.835893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.836011 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.836137 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.837006 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48056: BlockingRecv error: recv got EOF from 127.0.0.1:48056 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:39:06.878705 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.872656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.22.221.125:43401)
0926 07:39:06.872926 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.872933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.873317 (+ 384us) server_negotiation.cc:407] Connection header received
0926 07:39:06.873490 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.873498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.873597 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.873728 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.874125 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.875321 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.876665 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.877371 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.877513 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.877896 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.878040 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.878139 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.878287 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.878371 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 07:39:06.886951 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.884826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.22.221.124:35391)
0926 07:39:06.885090 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.885097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.885447 (+ 350us) server_negotiation.cc:407] Connection header received
0926 07:39:06.885649 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.885654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.885722 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.885801 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.886733 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48058: BlockingRecv error: recv got EOF from 127.0.0.1:48058 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:39:06.960661 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:06.954741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.22.221.125:43401)
0926 07:39:06.954979 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.954988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.955557 (+ 569us) server_negotiation.cc:407] Connection header received
0926 07:39:06.955791 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.955796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.955864 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.955966 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:06.956406 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.957340 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.958821 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:06.959337 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:06.959522 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:06.959945 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:06.960048 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:06.960150 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:06.960294 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 07:39:06.960372 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 07:39:06.970108 13936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:39:06.968167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.22.221.124:35391)
0926 07:39:06.968418 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:06.968426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:06.968681 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:39:06.968902 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:06.968908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:06.968983 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:06.969093 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:39:06.969902 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48066: BlockingRecv error: recv got EOF from 127.0.0.1:48066 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:39:07.035794 13950 catalog_manager.cc:1259] Loaded cluster ID: 681c42b8e7a4468e99755100569992ce
I20260926 07:39:07.035944 13950 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: loading cluster ID for follower catalog manager: success
I20260926 07:39:07.038710 13950 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: acquiring CA information for follower catalog manager: success
I20260926 07:39:07.040542 13950 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae8a08f82820458cb25b2b38aa49b931: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:39:07.107215 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:07.102088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.22.221.125:43401)
0926 07:39:07.102328 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:07.102334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:07.102607 (+ 273us) server_negotiation.cc:407] Connection header received
0926 07:39:07.102786 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:07.102791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:07.102872 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:07.102955 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:07.103421 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.104176 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.105596 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.105995 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.106087 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:07.106510 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:07.106611 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:07.106681 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:07.106812 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:39:07.106873 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 07:39:07.118944 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:07.113200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.22.221.124:35391)
0926 07:39:07.113413 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:07.113422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:07.113786 (+ 364us) server_negotiation.cc:407] Connection header received
0926 07:39:07.114003 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:07.114009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:07.114162 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:07.114283 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:07.114696 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.115449 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.116762 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.117173 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.117303 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:07.117811 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:07.118334 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:07.118489 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:07.118602 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:39:07.118688 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:39:07.133137 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:07.126004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.22.221.124:35391)
0926 07:39:07.126210 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:07.126219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:07.127292 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 07:39:07.127490 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:07.127495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:07.127570 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:07.127618 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:07.128432 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.129466 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.131292 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.131773 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.131906 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:07.132433 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:07.132555 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:07.132659 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:07.132774 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:39:07.132860 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:39:07.133490 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:07.125066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.22.221.125:43401)
0926 07:39:07.125292 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:07.125301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:07.126750 (+ 1449us) server_negotiation.cc:407] Connection header received
0926 07:39:07.127139 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:07.127144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:07.127268 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:07.127351 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:07.128008 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.128894 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.131046 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.131791 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.131915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:07.132587 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:07.132740 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:07.132821 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:07.133137 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:39:07.133221 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:39:07.134231 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:39:07.124933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.22.221.126:35561)
0926 07:39:07.126825 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0926 07:39:07.126833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:39:07.126850 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:39:07.126969 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:39:07.126978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:39:07.127202 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:39:07.128042 (+ 840us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:39:07.128197 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.129265 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.131476 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:39:07.132118 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:39:07.132240 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:39:07.132833 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:39:07.133306 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:39:07.133779 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0926 07:39:07.133831 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 07:39:07.133911 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1764,"thread_start_us":1661,"threads_started":1}
I20260926 07:39:07.142478 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13718
I20260926 07:39:07.143607 13780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:39:07.253362 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13718
I20260926 07:39:07.274871 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13792
I20260926 07:39:07.276124 13854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:39:07.384871 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13792
I20260926 07:39:07.405386 23413 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu pid 13866
I20260926 07:39:07.406487 13928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:39:07.514161 23413 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdo7xD1/build/asan/bin/kudu with pid 13866
2026-09-26T07:39:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4424 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8885 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358466 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST